# til

path: svg_dynamic-line-chart.md
topic: svg
title: Creating a dynamic line chart with SVG
url: https://github.com/simonw/til/blob/main/svg/dynamic-line-chart.md
created: 2020-08-22T11:34:56-07:00
created_utc: 2020-08-22T18:34:56+00:00
updated: 2020-08-22T22:24:22-07:00
updated_utc: 2020-08-23T05:24:22+00:00
shot_hash: 215c0260740169bb0575778f8f5f7533
slug: dynamic-line-chart
body:
I helped build the tide chart visualizations for [Rocky Beaches](https://www.rockybeaches.com/).

I wanted to generate an SVG line representing 24 hours of tide levels. I had 240 points per day of level data, imported from the NOAA Tides & Currents API. That data looked [like this](https://www.rockybeaches.com/data/tide_predictions):

| station_id | datetime | mllw_feet |
| --- | --- | --- |
| 9414131 | 2020-08-19 00:00 | 5.913 |
| 9414131 | 2020-08-19 00:06 | 5.822 |
| 9414131 | 2020-08-19 00:12 | 5.726 |
| 9414131 | 2020-08-19 00:18 | 5.623 |
| 9414131 | 2020-08-19 ... | ... |

I started with [this line chart example](https://css-tricks.com/how-to-make-charts-with-svg/#line-charts) from CSS-Tricks:

```svg
<svg viewBox="0 0 500 100">
  <polyline
     fill="none"
     stroke="#0074d9"
     stroke-width="3"
     points="
       0,120
       20,60
       40,80
       60,20"/>
</svg>
```
A few things to note:

- The SVG coordinate system starts at the top left - so '0,120' means 0 units along, 120 units down.
- The `viewBox` attribute specifies `min-x min-y width height` - so `0 0 500 100` specifies a box that is 500 units wide and 100 units high.

The dynamic data can be contained entirely in that single `points=` attribute on the `polyline`.

## Prototyping using SQL

I posted my working [in this issue](https://github.com/natbat/rockybeaches/issues/31), including prototyping the polyline using a SQL query that generated the `x1,y1 x2,y2` pairs. That query looked like this:

```sql
with today_points as (
  select
    datetime,
    mllw_feet
  from
    tide_predictions
  where
    date("datetime") = :p0
    and "station_id" = :p1
),
min_max as (
  select
    min(mllw_feet) as min_feet,
    max(mllw_feet) as max_feet
  from
    today_points
),
points as (
  select
    RANK () OVER (
      ORDER BY
        datetime
    ) -1 as rank,
    min_max.min_feet,
    min_max.max_feet,
    mllw_feet,
    (
      100 * (mllw_feet - min_max.min_feet) / (min_max.max_feet - min_max.min_feet)
    ) as line_height_pct
  from
    tide_predictions,
    min_max
  where
    date("datetime") = :p0
    and "station_id" = :p1
  order by
    datetime
)
select
  group_concat(rank || ',' || line_height_pct, ' ')
from
  points
```
[Try that here](https://www.rockybeaches.com/data?sql=with+today_points+as+%28%0D%0A++select+datetime%2C+mllw_feet%0D%0A++from+tide_predictions%0D%0A++where%0D%0A++date%28%22datetime%22%29+%3D+%3Ap0%0D%0A++and+%22station_id%22+%3D+%3Ap1%0D%0A%29%2C+min_max+as+%28select+min%28mllw_feet%29+as+min_feet%2C+max%28mllw_feet%29+as+max_feet+from+today_points%29%2C+points+as+%28select%0D%0A++RANK+%28%29+OVER+%28%0D%0A++++ORDER+BY%0D%0A++++++datetime%0D%0A++%29+-1+as+rank%2C%0D%0A+min_max.min_feet%2C+min_max.max_feet%2C++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++mllw_feet%2C%0D%0A++%28100+*+%28mllw_feet+-+min_max.min_feet%29+%2F+%28min_max.max_feet+-+min_max.min_feet%29%29+as+line_height_pct%0D%0Afrom%0D%0A++tide_predictions%2C+min_max%0D%0Awhere%0D%0A++date%28%22datetime%22%29+%3D+%3Ap0%0D%0A++and+%22station_id%22+%3D+%3Ap1%0D%0Aorder+by%0D%0A++datetime%29%0D%0Aselect+group_concat%28rank+%7C%7C+%27%2C%27+%7C%7C+line_height_pct%2C+%27+%27%29+from+points&p0=2020-08-21&p1=9414131)

I then pasted the results into the SVG and previewed it by pasting it into https://htmledit.squarefree.com/

```html
<svg style="border: 1px solid red; width: 40%; height: 60px" viewBox="0 -2 240 104" preserveAspectRatio="none">
  <polyline
     fill="none"
     stroke="#0074d9"
     stroke-width="2"
     points="0,0.543825975687781 1,0.255918106206011 2,0.0799744081893721 3,1.4210854715202e-14 4,0.0159948816378659 5,0.159948816378744 6,0.383877159309023 7,0.71976967370442 8,1.16762635956493 9,1.71145233525272 10,2.36724248240563 11,3.11900191938578 12,3.98272552783109 13,4.94241842610363 14,5.99808061420346 15,7.16570697376839 16,8.4133077415227 17,9.75687779910427 18,11.1804222648752 19,12.6999360204734 20,14.3154190658989 21,15.9948816378759 22,17.7543186180422 23,19.5937300063979 24,21.4971209213052 25,23.4644913627639 26,25.4958413307741 27,27.5911708253359 28,29.7344849648113 29,31.9097888675624 30,34.149072296865 31,36.4203454894434 32,38.7236084452975 33,41.0428662827895 34,43.3941138835573 35,45.7613563659629 36,48.1445937300064 37,50.5278310940499 38,52.9270633397313 39,55.3103007037748 40,57.6775431861804 41,60.0287907869482 42,62.3640435060781 43,64.6673064619322 44,66.9385796545106 45,69.1618682021753 46,71.3371721049264 47,73.4804862444018 48,75.5598208573257 49,77.575175943698 50,79.5265515035189 51,81.4139475367882 52,83.2213691618682 53,84.9488163787588 54,86.59628918746 55,88.1637875879719 56,89.6513115802943 57,91.0268714011516 58,92.3224568138196 59,93.5220729366603 60,94.6257197696737 61,95.6333973128599 62,96.5291106845809 63,97.3288547664747 64,98.0326295585413 65,98.6244401791427 66,99.104286628279 67,99.488163787588 68,99.7600767754319 69,99.9360204734485 70,100.0 71,99.9520153550864 72,99.7920665387076 73,99.5361484325016 74,99.1682661548304 75,98.6884197056942 76,98.0966090850928 77,97.4088291746641 78,96.6250799744082 79,95.7293666026871 80,94.7376839411388 81,93.6340371081254 82,92.4344209852847 83,91.1548304542546 84,89.7792706333973 85,88.3077415227127 86,86.7562380038388 87,85.1247600767754 88,83.4133077415227 89,81.6218809980806 90,79.7824696097249 91,77.8630838131798 92,75.9117082533589 93,73.8963531669866 94,71.8330134357006 95,69.7376839411388 96,67.5943698016635 97,65.4350607805502 98,63.2597568777991 99,61.0684580934101 100,58.8611644273832 101,56.6538707613564 102,54.4625719769674 103,52.2552783109405 104,50.0799744081894 105,47.9206653870761 106,45.7773512476008 107,43.6660268714011 108,41.5866922584773 109,39.5393474088292 110,37.5399872040947 111,35.5886116442738 112,33.6692258477287 113,31.829814459373 114,30.022392834293 115,28.2949456174024 116,26.6154830454255 117,25.0159948816379 118,23.4804862444018 119,22.0089571337172 120,20.6333973128599 121,19.3218170185541 122,18.1062060140755 123,16.9705694177863 124,15.9149072296865 125,14.9552143314139 126,14.0914907229686 127,13.3237364043506 128,12.635956493922 129,12.0601407549584 130,11.5802943058221 131,11.212412028151 132,10.9245041586692 133,10.7485604606526 134,10.6685860524632 135,10.7005758157389 136,10.828534868842 137,11.0524632117722 138,11.3723608445297 139,11.7882277671145 140,12.3000639795265 141,12.9078694817658 142,13.5956493921945 143,14.3793985924504 144,15.2431222008957 145,16.2028150991683 146,17.2264875239923 147,18.3301343570058 148,19.4977607165707 149,20.745361484325 150,22.0409468969929 151,23.4005118362124 152,24.8240563019834 153,26.2955854126679 154,27.7991042866283 155,29.3666026871401 156,30.9660908509277 157,32.5815738963532 158,34.2450415866923 159,35.9245041586692 160,37.6199616122841 161,39.3474088291747 162,41.0588611644274 163,42.786308381318 164,44.5137555982086 165,46.2412028150992 166,47.9526551503519 167,49.6481126039667 168,51.3275751759437 169,52.9750479846449 170,54.5905310300704 171,56.1740243122201 172,57.725527831094 173,59.2130518234165 174,60.6685860524632 175,62.0761356365963 176,63.40371081254 177,64.6992962252079 178,65.9149072296865 179,67.0665387076136 180,68.1541906589891 181,69.1618682021753 182,70.10556621881 183,70.9692898272553 184,71.7530390275112 185,72.4408189379399 186,73.064619321817 187,73.6084452975048 188,74.0563019833653 189,74.4241842610365 190,74.7120921305182 191,74.9040307101727 192,75.0159948816379 193,75.0319897632758 194,74.9520153550864 195,74.8080614203455 196,74.5521433141395 197,74.2162507997441 198,73.8003838771593 199,73.2885476647473 200,72.680742162508 201,72.0089571337172 202,71.2412028150992 203,70.3774792066539 204,69.4497760716571 205,68.4420985284709 206,67.3544465770953 207,66.1868202175304 208,64.9552143314139 209,63.6436340371081 210,62.2840690978887 211,60.8445297504799 212,59.3730006397953 213,57.8214971209213 214,56.2380038387716 215,54.6225207933461 216,52.9430582213691 217,51.2476007677543 218,49.5201535508637 219,47.7767114523352 220,46.0172744721689 221,44.2418426103647 222,42.4664107485605 223,40.6749840051184 224,38.8995521433141 225,37.1401151631478 226,35.3806781829814 227,33.6532309660909 228,31.9417786308381 229,30.2623160588612 230,28.61484325016 231,26.9993602047345 232,25.4318618042226 233,23.9123480486244 234,22.424824056302 235,21.001279590531 236,19.6257197696737 237,18.3141394753679 238,17.0665387076136 239,15.8829174664107"/>
</svg>
```
## Scaling the image with preserveAspectRatio="none"

I wanted to scale the SVG image to fit the div it was overlayed onto - which had a fixed height of 60px but a variable width.

[How to Scale SVG](https://css-tricks.com/scale-svg/) was useful for understanding the options here.

SVG images maintain their aspect ratio by default. Adding `preserveAspectRatio="none"` changed this, so the image could be distorted to fit the shape.

Unfortunately this has a visual impact on the line of the graph - causing it to be wider and narrower at different angles. We decided that this brush-stroke effect was actually OK for our purposes.

<img src="https://user-images.githubusercontent.com/9599/90821264-7a930680-e2e7-11ea-83a2-3003cf08877f.png" alt="Distorted line">

## Tweaking the viewBox

My chart was drawn be 240 units wide (covering 24 hours of the day with 10 units per hour) and 100 units high.

Using this as the exact viewBox dimensions caused the top and bottom of the line peaks to be truncated. I fixed that by tweaking the viewBox a bit to add some breathing room:

    viewBox="0 -2 240 104"

## Generating the data using Python

Here's [the final Python code](https://github.com/natbat/rockybeaches/blob/70039f18b3d3823a4f069deca513e950a3aaba4f/plugins/template_vars.py#L148-L156) I used to generate the points:

```python
# Calculate SVG points, refs https://github.com/natbat/rockybeaches/issues/31
min_feet = min(h["feet"] for h in heights[1:-1])
max_feet = max(h["feet"] for h in heights[1:-1])
feet_delta = max_feet - min_feet
svg_points = []
for i, height in enumerate(heights[1:-1]):
    ratio = (height["feet"] - min_feet) / feet_delta
    line_height_pct = 100 - (ratio * 100)
    svg_points.append((i, line_height_pct))
# ...
points = " ".join("{},{:.2f}".format(i, pct) for i, pct in svg_points)
```
I used `{:.2f}` to truncate the floating point represention to just two decimal places, which knocked 100KB off the total page size due to the number of charts being displayed!

Finished result can be seen here: https://www.rockybeaches.com/us/pillar-point#when-to-visit

html:
<p>I helped build the tide chart visualizations for <a href="https://www.rockybeaches.com/" rel="nofollow">Rocky Beaches</a>.</p>
<p>I wanted to generate an SVG line representing 24 hours of tide levels. I had 240 points per day of level data, imported from the NOAA Tides &amp; Currents API. That data looked <a href="https://www.rockybeaches.com/data/tide_predictions" rel="nofollow">like this</a>:</p>
<table>
<thead>
<tr>
<th>station_id</th>
<th>datetime</th>
<th>mllw_feet</th>
</tr>
</thead>
<tbody>
<tr>
<td>9414131</td>
<td>2020-08-19 00:00</td>
<td>5.913</td>
</tr>
<tr>
<td>9414131</td>
<td>2020-08-19 00:06</td>
<td>5.822</td>
</tr>
<tr>
<td>9414131</td>
<td>2020-08-19 00:12</td>
<td>5.726</td>
</tr>
<tr>
<td>9414131</td>
<td>2020-08-19 00:18</td>
<td>5.623</td>
</tr>
<tr>
<td>9414131</td>
<td>2020-08-19 ...</td>
<td>...</td>
</tr>
</tbody>
</table>
<p>I started with <a href="https://css-tricks.com/how-to-make-charts-with-svg/#line-charts" rel="nofollow">this line chart example</a> from CSS-Tricks:</p>
<div class="highlight highlight-text-xml-svg"><pre>&lt;<span class="pl-ent">svg</span> <span class="pl-e">viewBox</span>=<span class="pl-s"><span class="pl-pds">"</span>0 0 500 100<span class="pl-pds">"</span></span>&gt;
  &lt;<span class="pl-ent">polyline</span>
     <span class="pl-e">fill</span>=<span class="pl-s"><span class="pl-pds">"</span>none<span class="pl-pds">"</span></span>
     <span class="pl-e">stroke</span>=<span class="pl-s"><span class="pl-pds">"</span>#0074d9<span class="pl-pds">"</span></span>
     <span class="pl-e">stroke-width</span>=<span class="pl-s"><span class="pl-pds">"</span>3<span class="pl-pds">"</span></span>
     <span class="pl-e">points</span>=<span class="pl-s"><span class="pl-pds">"</span></span>
<span class="pl-s">       0,120</span>
<span class="pl-s">       20,60</span>
<span class="pl-s">       40,80</span>
<span class="pl-s">       60,20<span class="pl-pds">"</span></span>/&gt;
&lt;/<span class="pl-ent">svg</span>&gt;</pre></div>
<p>A few things to note:</p>
<ul>
<li>The SVG coordinate system starts at the top left - so '0,120' means 0 units along, 120 units down.</li>
<li>The <code>viewBox</code> attribute specifies <code>min-x min-y width height</code> - so <code>0 0 500 100</code> specifies a box that is 500 units wide and 100 units high.</li>
</ul>
<p>The dynamic data can be contained entirely in that single <code>points=</code> attribute on the <code>polyline</code>.</p>
<h2>
<a id="user-content-prototyping-using-sql" class="anchor" href="#prototyping-using-sql" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Prototyping using SQL</h2>
<p>I posted my working <a href="https://github.com/natbat/rockybeaches/issues/31">in this issue</a>, including prototyping the polyline using a SQL query that generated the <code>x1,y1 x2,y2</code> pairs. That query looked like this:</p>
<div class="highlight highlight-source-sql"><pre>with today_points <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    datetime,
    mllw_feet
  <span class="pl-k">from</span>
    tide_predictions
  <span class="pl-k">where</span>
    <span class="pl-k">date</span>(<span class="pl-s"><span class="pl-pds">"</span>datetime<span class="pl-pds">"</span></span>) <span class="pl-k">=</span> :p0
    <span class="pl-k">and</span> <span class="pl-s"><span class="pl-pds">"</span>station_id<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :p1
),
min_max <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    <span class="pl-c1">min</span>(mllw_feet) <span class="pl-k">as</span> min_feet,
    <span class="pl-c1">max</span>(mllw_feet) <span class="pl-k">as</span> max_feet
  <span class="pl-k">from</span>
    today_points
),
points <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    RANK () OVER (
      <span class="pl-k">ORDER BY</span>
        datetime
    ) <span class="pl-k">-</span><span class="pl-c1">1</span> <span class="pl-k">as</span> rank,
    <span class="pl-c1">min_max</span>.<span class="pl-c1">min_feet</span>,
    <span class="pl-c1">min_max</span>.<span class="pl-c1">max_feet</span>,
    mllw_feet,
    (
      <span class="pl-c1">100</span> <span class="pl-k">*</span> (mllw_feet <span class="pl-k">-</span> <span class="pl-c1">min_max</span>.<span class="pl-c1">min_feet</span>) <span class="pl-k">/</span> (<span class="pl-c1">min_max</span>.<span class="pl-c1">max_feet</span> <span class="pl-k">-</span> <span class="pl-c1">min_max</span>.<span class="pl-c1">min_feet</span>)
    ) <span class="pl-k">as</span> line_height_pct
  <span class="pl-k">from</span>
    tide_predictions,
    min_max
  <span class="pl-k">where</span>
    <span class="pl-k">date</span>(<span class="pl-s"><span class="pl-pds">"</span>datetime<span class="pl-pds">"</span></span>) <span class="pl-k">=</span> :p0
    <span class="pl-k">and</span> <span class="pl-s"><span class="pl-pds">"</span>station_id<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :p1
  <span class="pl-k">order by</span>
    datetime
)
<span class="pl-k">select</span>
  group_concat(rank <span class="pl-k">||</span> <span class="pl-s"><span class="pl-pds">'</span>,<span class="pl-pds">'</span></span> <span class="pl-k">||</span> line_height_pct, <span class="pl-s"><span class="pl-pds">'</span> <span class="pl-pds">'</span></span>)
<span class="pl-k">from</span>
  points</pre></div>
<p><a href="https://www.rockybeaches.com/data?sql=with+today_points+as+%28%0D%0A++select+datetime%2C+mllw_feet%0D%0A++from+tide_predictions%0D%0A++where%0D%0A++date%28%22datetime%22%29+%3D+%3Ap0%0D%0A++and+%22station_id%22+%3D+%3Ap1%0D%0A%29%2C+min_max+as+%28select+min%28mllw_feet%29+as+min_feet%2C+max%28mllw_feet%29+as+max_feet+from+today_points%29%2C+points+as+%28select%0D%0A++RANK+%28%29+OVER+%28%0D%0A++++ORDER+BY%0D%0A++++++datetime%0D%0A++%29+-1+as+rank%2C%0D%0A+min_max.min_feet%2C+min_max.max_feet%2C++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++++mllw_feet%2C%0D%0A++%28100+*+%28mllw_feet+-+min_max.min_feet%29+%2F+%28min_max.max_feet+-+min_max.min_feet%29%29+as+line_height_pct%0D%0Afrom%0D%0A++tide_predictions%2C+min_max%0D%0Awhere%0D%0A++date%28%22datetime%22%29+%3D+%3Ap0%0D%0A++and+%22station_id%22+%3D+%3Ap1%0D%0Aorder+by%0D%0A++datetime%29%0D%0Aselect+group_concat%28rank+%7C%7C+%27%2C%27+%7C%7C+line_height_pct%2C+%27+%27%29+from+points&amp;p0=2020-08-21&amp;p1=9414131" rel="nofollow">Try that here</a></p>
<p>I then pasted the results into the SVG and previewed it by pasting it into <a href="https://htmledit.squarefree.com/" rel="nofollow">https://htmledit.squarefree.com/</a></p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">svg</span> <span class="pl-c1">style</span>="<span class="pl-s">border: 1px solid red; width: 40%; height: 60px</span>" <span class="pl-c1">viewBox</span>="<span class="pl-s">0 -2 240 104</span>" <span class="pl-c1">preserveAspectRatio</span>="<span class="pl-s">none</span>"<span class="pl-kos">&gt;</span>
  <span class="pl-kos">&lt;</span><span class="pl-ent">polyline</span>
     <span class="pl-c1">fill</span>="<span class="pl-s">none</span>"
     <span class="pl-c1">stroke</span>="<span class="pl-s">#0074d9</span>"
     <span class="pl-c1">stroke-width</span>="<span class="pl-s">2</span>"
     <span class="pl-c1">points</span>="<span class="pl-s">0,0.543825975687781 1,0.255918106206011 2,0.0799744081893721 3,1.4210854715202e-14 4,0.0159948816378659 5,0.159948816378744 6,0.383877159309023 7,0.71976967370442 8,1.16762635956493 9,1.71145233525272 10,2.36724248240563 11,3.11900191938578 12,3.98272552783109 13,4.94241842610363 14,5.99808061420346 15,7.16570697376839 16,8.4133077415227 17,9.75687779910427 18,11.1804222648752 19,12.6999360204734 20,14.3154190658989 21,15.9948816378759 22,17.7543186180422 23,19.5937300063979 24,21.4971209213052 25,23.4644913627639 26,25.4958413307741 27,27.5911708253359 28,29.7344849648113 29,31.9097888675624 30,34.149072296865 31,36.4203454894434 32,38.7236084452975 33,41.0428662827895 34,43.3941138835573 35,45.7613563659629 36,48.1445937300064 37,50.5278310940499 38,52.9270633397313 39,55.3103007037748 40,57.6775431861804 41,60.0287907869482 42,62.3640435060781 43,64.6673064619322 44,66.9385796545106 45,69.1618682021753 46,71.3371721049264 47,73.4804862444018 48,75.5598208573257 49,77.575175943698 50,79.5265515035189 51,81.4139475367882 52,83.2213691618682 53,84.9488163787588 54,86.59628918746 55,88.1637875879719 56,89.6513115802943 57,91.0268714011516 58,92.3224568138196 59,93.5220729366603 60,94.6257197696737 61,95.6333973128599 62,96.5291106845809 63,97.3288547664747 64,98.0326295585413 65,98.6244401791427 66,99.104286628279 67,99.488163787588 68,99.7600767754319 69,99.9360204734485 70,100.0 71,99.9520153550864 72,99.7920665387076 73,99.5361484325016 74,99.1682661548304 75,98.6884197056942 76,98.0966090850928 77,97.4088291746641 78,96.6250799744082 79,95.7293666026871 80,94.7376839411388 81,93.6340371081254 82,92.4344209852847 83,91.1548304542546 84,89.7792706333973 85,88.3077415227127 86,86.7562380038388 87,85.1247600767754 88,83.4133077415227 89,81.6218809980806 90,79.7824696097249 91,77.8630838131798 92,75.9117082533589 93,73.8963531669866 94,71.8330134357006 95,69.7376839411388 96,67.5943698016635 97,65.4350607805502 98,63.2597568777991 99,61.0684580934101 100,58.8611644273832 101,56.6538707613564 102,54.4625719769674 103,52.2552783109405 104,50.0799744081894 105,47.9206653870761 106,45.7773512476008 107,43.6660268714011 108,41.5866922584773 109,39.5393474088292 110,37.5399872040947 111,35.5886116442738 112,33.6692258477287 113,31.829814459373 114,30.022392834293 115,28.2949456174024 116,26.6154830454255 117,25.0159948816379 118,23.4804862444018 119,22.0089571337172 120,20.6333973128599 121,19.3218170185541 122,18.1062060140755 123,16.9705694177863 124,15.9149072296865 125,14.9552143314139 126,14.0914907229686 127,13.3237364043506 128,12.635956493922 129,12.0601407549584 130,11.5802943058221 131,11.212412028151 132,10.9245041586692 133,10.7485604606526 134,10.6685860524632 135,10.7005758157389 136,10.828534868842 137,11.0524632117722 138,11.3723608445297 139,11.7882277671145 140,12.3000639795265 141,12.9078694817658 142,13.5956493921945 143,14.3793985924504 144,15.2431222008957 145,16.2028150991683 146,17.2264875239923 147,18.3301343570058 148,19.4977607165707 149,20.745361484325 150,22.0409468969929 151,23.4005118362124 152,24.8240563019834 153,26.2955854126679 154,27.7991042866283 155,29.3666026871401 156,30.9660908509277 157,32.5815738963532 158,34.2450415866923 159,35.9245041586692 160,37.6199616122841 161,39.3474088291747 162,41.0588611644274 163,42.786308381318 164,44.5137555982086 165,46.2412028150992 166,47.9526551503519 167,49.6481126039667 168,51.3275751759437 169,52.9750479846449 170,54.5905310300704 171,56.1740243122201 172,57.725527831094 173,59.2130518234165 174,60.6685860524632 175,62.0761356365963 176,63.40371081254 177,64.6992962252079 178,65.9149072296865 179,67.0665387076136 180,68.1541906589891 181,69.1618682021753 182,70.10556621881 183,70.9692898272553 184,71.7530390275112 185,72.4408189379399 186,73.064619321817 187,73.6084452975048 188,74.0563019833653 189,74.4241842610365 190,74.7120921305182 191,74.9040307101727 192,75.0159948816379 193,75.0319897632758 194,74.9520153550864 195,74.8080614203455 196,74.5521433141395 197,74.2162507997441 198,73.8003838771593 199,73.2885476647473 200,72.680742162508 201,72.0089571337172 202,71.2412028150992 203,70.3774792066539 204,69.4497760716571 205,68.4420985284709 206,67.3544465770953 207,66.1868202175304 208,64.9552143314139 209,63.6436340371081 210,62.2840690978887 211,60.8445297504799 212,59.3730006397953 213,57.8214971209213 214,56.2380038387716 215,54.6225207933461 216,52.9430582213691 217,51.2476007677543 218,49.5201535508637 219,47.7767114523352 220,46.0172744721689 221,44.2418426103647 222,42.4664107485605 223,40.6749840051184 224,38.8995521433141 225,37.1401151631478 226,35.3806781829814 227,33.6532309660909 228,31.9417786308381 229,30.2623160588612 230,28.61484325016 231,26.9993602047345 232,25.4318618042226 233,23.9123480486244 234,22.424824056302 235,21.001279590531 236,19.6257197696737 237,18.3141394753679 238,17.0665387076136 239,15.8829174664107</span>"/&gt;
<span class="pl-kos">&lt;/</span><span class="pl-ent">svg</span><span class="pl-kos">&gt;</span></pre></div>
<h2>
<a id="user-content-scaling-the-image-with-preserveaspectrationone" class="anchor" href="#scaling-the-image-with-preserveaspectrationone" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Scaling the image with preserveAspectRatio="none"</h2>
<p>I wanted to scale the SVG image to fit the div it was overlayed onto - which had a fixed height of 60px but a variable width.</p>
<p><a href="https://css-tricks.com/scale-svg/" rel="nofollow">How to Scale SVG</a> was useful for understanding the options here.</p>
<p>SVG images maintain their aspect ratio by default. Adding <code>preserveAspectRatio="none"</code> changed this, so the image could be distorted to fit the shape.</p>
<p>Unfortunately this has a visual impact on the line of the graph - causing it to be wider and narrower at different angles. We decided that this brush-stroke effect was actually OK for our purposes.</p>
<p><a href="https://user-images.githubusercontent.com/9599/90821264-7a930680-e2e7-11ea-83a2-3003cf08877f.png" target="_blank" rel="nofollow"><img src="https://user-images.githubusercontent.com/9599/90821264-7a930680-e2e7-11ea-83a2-3003cf08877f.png" alt="Distorted line" style="max-width:100%;"></a></p>
<h2>
<a id="user-content-tweaking-the-viewbox" class="anchor" href="#tweaking-the-viewbox" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Tweaking the viewBox</h2>
<p>My chart was drawn be 240 units wide (covering 24 hours of the day with 10 units per hour) and 100 units high.</p>
<p>Using this as the exact viewBox dimensions caused the top and bottom of the line peaks to be truncated. I fixed that by tweaking the viewBox a bit to add some breathing room:</p>
<pre><code>viewBox="0 -2 240 104"
</code></pre>
<h2>
<a id="user-content-generating-the-data-using-python" class="anchor" href="#generating-the-data-using-python" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Generating the data using Python</h2>
<p>Here's <a href="https://github.com/natbat/rockybeaches/blob/70039f18b3d3823a4f069deca513e950a3aaba4f/plugins/template_vars.py#L148-L156">the final Python code</a> I used to generate the points:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-c"># Calculate SVG points, refs https://github.com/natbat/rockybeaches/issues/31</span>
<span class="pl-s1">min_feet</span> <span class="pl-c1">=</span> <span class="pl-en">min</span>(<span class="pl-s1">h</span>[<span class="pl-s">"feet"</span>] <span class="pl-k">for</span> <span class="pl-s1">h</span> <span class="pl-c1">in</span> <span class="pl-s1">heights</span>[<span class="pl-c1">1</span>:<span class="pl-c1">-</span><span class="pl-c1">1</span>])
<span class="pl-s1">max_feet</span> <span class="pl-c1">=</span> <span class="pl-en">max</span>(<span class="pl-s1">h</span>[<span class="pl-s">"feet"</span>] <span class="pl-k">for</span> <span class="pl-s1">h</span> <span class="pl-c1">in</span> <span class="pl-s1">heights</span>[<span class="pl-c1">1</span>:<span class="pl-c1">-</span><span class="pl-c1">1</span>])
<span class="pl-s1">feet_delta</span> <span class="pl-c1">=</span> <span class="pl-s1">max_feet</span> <span class="pl-c1">-</span> <span class="pl-s1">min_feet</span>
<span class="pl-s1">svg_points</span> <span class="pl-c1">=</span> []
<span class="pl-k">for</span> <span class="pl-s1">i</span>, <span class="pl-s1">height</span> <span class="pl-c1">in</span> <span class="pl-en">enumerate</span>(<span class="pl-s1">heights</span>[<span class="pl-c1">1</span>:<span class="pl-c1">-</span><span class="pl-c1">1</span>]):
    <span class="pl-s1">ratio</span> <span class="pl-c1">=</span> (<span class="pl-s1">height</span>[<span class="pl-s">"feet"</span>] <span class="pl-c1">-</span> <span class="pl-s1">min_feet</span>) <span class="pl-c1">/</span> <span class="pl-s1">feet_delta</span>
    <span class="pl-s1">line_height_pct</span> <span class="pl-c1">=</span> <span class="pl-c1">100</span> <span class="pl-c1">-</span> (<span class="pl-s1">ratio</span> <span class="pl-c1">*</span> <span class="pl-c1">100</span>)
    <span class="pl-s1">svg_points</span>.<span class="pl-en">append</span>((<span class="pl-s1">i</span>, <span class="pl-s1">line_height_pct</span>))
<span class="pl-c"># ...</span>
<span class="pl-s1">points</span> <span class="pl-c1">=</span> <span class="pl-s">" "</span>.<span class="pl-en">join</span>(<span class="pl-s">"{},{:.2f}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">i</span>, <span class="pl-s1">pct</span>) <span class="pl-k">for</span> <span class="pl-s1">i</span>, <span class="pl-s1">pct</span> <span class="pl-c1">in</span> <span class="pl-s1">svg_points</span>)</pre></div>
<p>I used <code>{:.2f}</code> to truncate the floating point represention to just two decimal places, which knocked 100KB off the total page size due to the number of charts being displayed!</p>
<p>Finished result can be seen here: <a href="https://www.rockybeaches.com/us/pillar-point#when-to-visit" rel="nofollow">https://www.rockybeaches.com/us/pillar-point#when-to-visit</a></p>


path: firefox_search-across-all-resources.md
topic: firefox
title: Search across all loaded resources in Firefox
url: https://github.com/simonw/til/blob/main/firefox/search-across-all-resources.md
created: 2020-05-05T11:38:56-07:00
created_utc: 2020-05-05T18:38:56+00:00
updated: 2020-09-20T21:43:17-07:00
updated_utc: 2020-09-21T04:43:17+00:00
shot_hash: 0cf1e455f161435a4aea07480c27da89
slug: search-across-all-resources
body:
You can search for a string in any resource loaded by a page (including across HTML, JavaScript and CSS) in the Debugger pane by hitting Command+Shift+F.

<img alt="Screenshot of search interface" src="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources.jpg" width="600">

This view doesn't search the body of any JSON assets that were fetched by code, presumably because JSON isn't automatically loaded into memory by the browser.

But ([thanks, @digitarald](https://twitter.com/digitarald/status/1257748744352567296)) the Network pane DOES let you search for content in assets fetched via Ajax/fetch() etc - though you do have to run the search before you trigger the requests that the search should cover. Again, the shortcut is Command+Shift+F.

<img alt="Screenshot of search interface" src="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources-2.jpg" width="600">

html:
<p>You can search for a string in any resource loaded by a page (including across HTML, JavaScript and CSS) in the Debugger pane by hitting Command+Shift+F.</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources.jpg" target="_blank" rel="nofollow"><img alt="Screenshot of search interface" src="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources.jpg" width="600" style="max-width:100%;"></a></p>
<p>This view doesn't search the body of any JSON assets that were fetched by code, presumably because JSON isn't automatically loaded into memory by the browser.</p>
<p>But (<a href="https://twitter.com/digitarald/status/1257748744352567296" rel="nofollow">thanks, @digitarald</a>) the Network pane DOES let you search for content in assets fetched via Ajax/fetch() etc - though you do have to run the search before you trigger the requests that the search should cover. Again, the shortcut is Command+Shift+F.</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources-2.jpg" target="_blank" rel="nofollow"><img alt="Screenshot of search interface" src="https://raw.githubusercontent.com/simonw/til/main/firefox/search-across-all-resources-2.jpg" width="600" style="max-width:100%;"></a></p>


path: asgi_lifespan-test-httpx.md
topic: asgi
title: Writing tests for the ASGI lifespan protocol with HTTPX
url: https://github.com/simonw/til/blob/main/asgi/lifespan-test-httpx.md
created: 2020-06-29T09:13:49-07:00
created_utc: 2020-06-29T16:13:49+00:00
updated: 2020-07-06T13:46:05-07:00
updated_utc: 2020-07-06T20:46:05+00:00
shot_hash: 4656e3d750ad850e94a3240ebcbcbb26
slug: lifespan-test-httpx
body:
Uvicorn silently ignores exceptions that occur during startup against the ASGI lifespan protocol - see [starlette/issues/486](https://github.com/encode/starlette/issues/486).

You can disable this feature using the `lifespan="on"` parameter to `uvicorn.run()` - which Datasette now does as-of [16f592247a2a0e140ada487e9972645406dcae69](https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69)

This exposed a bug in `datasette-debug-asgi`: it wasn't handling lifespan events correctly. [datasette-debug-asgi/issues/1](https://github.com/simonw/datasette-debug-asgi/issues/1)

The unit tests weren't catching this because using HTTPX to make test requests [doesn't trigger lifespan events](https://github.com/encode/httpx/issues/350).

Florimond Manca had run into this problem too, and built [asgi-lifespan](https://github.com/florimondmanca/asgi-lifespan) to address it.

You can wrap an ASGI app in `async with LifespanManager(app):` and the correct lifespan events will be fired by that with block.

Here's how to use it to [trigger lifespan events in a test](https://github.com/simonw/datasette-debug-asgi/blob/72d568d32a3159c763ce908c0b269736935c6987/test_datasette_debug_asgi.py):

```python
from asgi_lifespan import LifespanManager

@pytest.mark.asyncio
async def test_datasette_debug_asgi():
    ds = Datasette([], memory=True)
    app = ds.app()
    async with LifespanManager(app):
        async with httpx.AsyncClient(app=app) as client:
            response = await client.get("http://localhost/-/asgi-scope")
            assert 200 == response.status_code
            assert "text/plain; charset=UTF-8" == response.headers["content-type"]
```

html:
<p>Uvicorn silently ignores exceptions that occur during startup against the ASGI lifespan protocol - see <a href="https://github.com/encode/starlette/issues/486">starlette/issues/486</a>.</p>
<p>You can disable this feature using the <code>lifespan="on"</code> parameter to <code>uvicorn.run()</code> - which Datasette now does as-of <a href="https://github.com/simonw/datasette/commit/16f592247a2a0e140ada487e9972645406dcae69">16f592247a2a0e140ada487e9972645406dcae69</a></p>
<p>This exposed a bug in <code>datasette-debug-asgi</code>: it wasn't handling lifespan events correctly. <a href="https://github.com/simonw/datasette-debug-asgi/issues/1">datasette-debug-asgi/issues/1</a></p>
<p>The unit tests weren't catching this because using HTTPX to make test requests <a href="https://github.com/encode/httpx/issues/350">doesn't trigger lifespan events</a>.</p>
<p>Florimond Manca had run into this problem too, and built <a href="https://github.com/florimondmanca/asgi-lifespan">asgi-lifespan</a> to address it.</p>
<p>You can wrap an ASGI app in <code>async with LifespanManager(app):</code> and the correct lifespan events will be fired by that with block.</p>
<p>Here's how to use it to <a href="https://github.com/simonw/datasette-debug-asgi/blob/72d568d32a3159c763ce908c0b269736935c6987/test_datasette_debug_asgi.py">trigger lifespan events in a test</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">asgi_lifespan</span> <span class="pl-k">import</span> <span class="pl-v">LifespanManager</span>

<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-s1">asyncio</span></span>
<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">test_datasette_debug_asgi</span>():
    <span class="pl-s1">ds</span> <span class="pl-c1">=</span> <span class="pl-v">Datasette</span>([], <span class="pl-s1">memory</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)
    <span class="pl-s1">app</span> <span class="pl-c1">=</span> <span class="pl-s1">ds</span>.<span class="pl-en">app</span>()
    <span class="pl-k">async</span> <span class="pl-k">with</span> <span class="pl-v">LifespanManager</span>(<span class="pl-s1">app</span>):
        <span class="pl-k">async</span> <span class="pl-k">with</span> <span class="pl-s1">httpx</span>.<span class="pl-v">AsyncClient</span>(<span class="pl-s1">app</span><span class="pl-c1">=</span><span class="pl-s1">app</span>) <span class="pl-k">as</span> <span class="pl-s1">client</span>:
            <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">client</span>.<span class="pl-en">get</span>(<span class="pl-s">"http://localhost/-/asgi-scope"</span>)
            <span class="pl-k">assert</span> <span class="pl-c1">200</span> <span class="pl-c1">==</span> <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span>
            <span class="pl-k">assert</span> <span class="pl-s">"text/plain; charset=UTF-8"</span> <span class="pl-c1">==</span> <span class="pl-s1">response</span>.<span class="pl-s1">headers</span>[<span class="pl-s">"content-type"</span>]</pre></div>


path: pypi_project-links.md
topic: pypi
title: Adding project links to PyPI
url: https://github.com/simonw/til/blob/main/pypi/project-links.md
created: 2020-05-11T09:33:05-07:00
created_utc: 2020-05-11T16:33:05+00:00
updated: 2020-09-01T16:54:36-07:00
updated_utc: 2020-09-01T23:54:36+00:00
shot_hash: 1447c8cdd4caa68e5514a1bb5b9f9f49
slug: project-links
body:
I spotted a neat looking project links section on https://pypi.org/project/ExifReader/

Turns out that's added using a `project_urls` key in setup.py, e.g. here: https://gitlab.com/Cyb3r-Jak3/exifreader/-/blob/publish/setup.py

```python
    project_urls={
        "Issues": "https://gitlab.com/Cyb3r-Jak3/ExifReader/issues",
        "Source Code": "https://gitlab.com/Cyb3r-Jak3/ExifReader/-/tree/publish",
        "CI": "https://gitlab.com/Cyb3r-Jak3/ExifReader/pipelines",
        "Releases": "https://github.com/Cyb3r-Jak3/ExifReader"
    },
```
The [documentation](https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls) says that the keys can be any string you like. So where do those icons come from?

I looked at the HTML source code and searched the PyPI website source code for one of the CSS classes I found there: https://github.com/pypa/warehouse/search?q=fa-bug&unscoped_q=fa-bug

This lead me to the macro that defines the custom icons, which shows what custom keys or URLs are supported: https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60

I added these to the [sqlite-utils PyPI page](https://pypi.org/project/sqlite-utils/) in [this commit](https://github.com/simonw/sqlite-utils/commit/74b30af31bf5169559c06aa6e57e1e4873076720).

<img src="https://raw.githubusercontent.com/simonw/til/main/pypi/project-links.png" width="333" alt="Project links on PyPI">

html:
<p>I spotted a neat looking project links section on <a href="https://pypi.org/project/ExifReader/" rel="nofollow">https://pypi.org/project/ExifReader/</a></p>
<p>Turns out that's added using a <code>project_urls</code> key in setup.py, e.g. here: <a href="https://gitlab.com/Cyb3r-Jak3/exifreader/-/blob/publish/setup.py" rel="nofollow">https://gitlab.com/Cyb3r-Jak3/exifreader/-/blob/publish/setup.py</a></p>
<div class="highlight highlight-source-python"><pre>    <span class="pl-s1">project_urls</span><span class="pl-c1">=</span>{
        <span class="pl-s">"Issues"</span>: <span class="pl-s">"https://gitlab.com/Cyb3r-Jak3/ExifReader/issues"</span>,
        <span class="pl-s">"Source Code"</span>: <span class="pl-s">"https://gitlab.com/Cyb3r-Jak3/ExifReader/-/tree/publish"</span>,
        <span class="pl-s">"CI"</span>: <span class="pl-s">"https://gitlab.com/Cyb3r-Jak3/ExifReader/pipelines"</span>,
        <span class="pl-s">"Releases"</span>: <span class="pl-s">"https://github.com/Cyb3r-Jak3/ExifReader"</span>
    },</pre></div>
<p>The <a href="https://packaging.python.org/guides/distributing-packages-using-setuptools/#project-urls" rel="nofollow">documentation</a> says that the keys can be any string you like. So where do those icons come from?</p>
<p>I looked at the HTML source code and searched the PyPI website source code for one of the CSS classes I found there: <a href="https://github.com/pypa/warehouse/search?q=fa-bug&amp;unscoped_q=fa-bug">https://github.com/pypa/warehouse/search?q=fa-bug&amp;unscoped_q=fa-bug</a></p>
<p>This lead me to the macro that defines the custom icons, which shows what custom keys or URLs are supported: <a href="https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60">https://github.com/pypa/warehouse/blob/2f00f4a9f208546ff0ebb6a6e61439021ca60a43/warehouse/templates/packaging/detail.html#L16-L60</a></p>
<p>I added these to the <a href="https://pypi.org/project/sqlite-utils/" rel="nofollow">sqlite-utils PyPI page</a> in <a href="https://github.com/simonw/sqlite-utils/commit/74b30af31bf5169559c06aa6e57e1e4873076720">this commit</a>.</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/pypi/project-links.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/pypi/project-links.png" width="333" alt="Project links on PyPI" style="max-width:100%;"></a></p>


path: docker_attach-bash-to-running-container.md
topic: docker
title: Attaching a bash shell to a running Docker container
url: https://github.com/simonw/til/blob/main/docker/attach-bash-to-running-container.md
created: 2020-08-10T08:41:01-07:00
created_utc: 2020-08-10T15:41:01+00:00
updated: 2020-08-10T08:41:01-07:00
updated_utc: 2020-08-10T15:41:01+00:00
shot_hash: 2bd34a112abf7858d6a88c4da46c3bda
slug: attach-bash-to-running-container
body:
Use `docker ps` to find the container ID:

    $ docker ps                        
    CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
    81b2ad3194cb        alexdebrie/livegrep-base:1   "/livegrep-github-re…"   2 minutes ago       Up 2 minutes                            compassionate_yalow

Run `docker exec -it ID bash` to start a bash session in that container:

    $ docker exec -it 81b2ad3194cb bash

I made the mistake of using `docker attach 81b2ad3194cb` first, which attaches you to the command running as CMD in that conatiner, and means that if you hit `Ctrl+C` you exit that command and terminate the container!

html:
<p>Use <code>docker ps</code> to find the container ID:</p>
<pre><code>$ docker ps                        
CONTAINER ID        IMAGE                        COMMAND                  CREATED             STATUS              PORTS               NAMES
81b2ad3194cb        alexdebrie/livegrep-base:1   "/livegrep-github-re…"   2 minutes ago       Up 2 minutes                            compassionate_yalow
</code></pre>
<p>Run <code>docker exec -it ID bash</code> to start a bash session in that container:</p>
<pre><code>$ docker exec -it 81b2ad3194cb bash
</code></pre>
<p>I made the mistake of using <code>docker attach 81b2ad3194cb</code> first, which attaches you to the command running as CMD in that conatiner, and means that if you hit <code>Ctrl+C</code> you exit that command and terminate the container!</p>


path: macos_skitch-catalina.md
topic: macos
title: Get Skitch working on Catalina
url: https://github.com/simonw/til/blob/main/macos/skitch-catalina.md
created: 2020-04-21T18:48:33-07:00
created_utc: 2020-04-22T01:48:33+00:00
updated: 2020-04-21T18:48:33-07:00
updated_utc: 2020-04-22T01:48:33+00:00
shot_hash: caa71dd7c7313d6df5fe76db376127c7
slug: skitch-catalina
body:
I really like Skitch for taking screeshots, mainly because I like to snap a quick shot and then drag the icon onto a Google Doc or similar without saving the file anywhere.

Getting it working on Catalina was a bit tricky.

First, install it from the App Store.

Next, disable the default Shift+Command+5 system shortcut so Skitch can use it instead:

![Keyboard shortcut settings](https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-1.png)

Then give Skitch permission to record your screen. Without that every screenshot will come out as a snippet of your desktop background!

![Privace -> screen recording settings](https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-2.png)

html:
<p>I really like Skitch for taking screeshots, mainly because I like to snap a quick shot and then drag the icon onto a Google Doc or similar without saving the file anywhere.</p>
<p>Getting it working on Catalina was a bit tricky.</p>
<p>First, install it from the App Store.</p>
<p>Next, disable the default Shift+Command+5 system shortcut so Skitch can use it instead:</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-1.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-1.png" alt="Keyboard shortcut settings" style="max-width:100%;"></a></p>
<p>Then give Skitch permission to record your screen. Without that every screenshot will come out as a snippet of your desktop background!</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-2.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/master/macos/skitch-catalina-2.png" alt="Privace -&gt; screen recording settings" style="max-width:100%;"></a></p>


path: macos_fixing-compinit-insecure-directories.md
topic: macos
title: Fixing "compinit: insecure directories" error
url: https://github.com/simonw/til/blob/main/macos/fixing-compinit-insecure-directories.md
created: 2020-04-26T09:49:58-07:00
created_utc: 2020-04-26T16:49:58+00:00
updated: 2020-04-26T09:49:58-07:00
updated_utc: 2020-04-26T16:49:58+00:00
shot_hash: 5df4197e594fa4db0129150d72a9f91c
slug: fixing-compinit-insecure-directories
body:
Every time I opened a terminal on my new Mac running Catalina with zsh I got the following annoying error:

```
Last login: Fri Apr 24 17:50:41 on ttys004
zsh compinit: insecure directories, run compaudit for list.
Ignore insecure directories and continue [y] or abort compinit [n]? ncompinit: initialization aborted
```

Based on a tip in https://github.com/zsh-users/zsh-completions/issues/680 I fixed it by doing this:

```
% compaudit
There are insecure directories:
/usr/local/share/zsh/site-functions
/usr/local/share/zsh
```

Then I ran these commands to fix it:

```
sudo chmod g-w /usr/local/share/zsh/site-functions
sudo chmod g-w /usr/local/share/zsh
```

html:
<p>Every time I opened a terminal on my new Mac running Catalina with zsh I got the following annoying error:</p>
<pre><code>Last login: Fri Apr 24 17:50:41 on ttys004
zsh compinit: insecure directories, run compaudit for list.
Ignore insecure directories and continue [y] or abort compinit [n]? ncompinit: initialization aborted
</code></pre>
<p>Based on a tip in <a href="https://github.com/zsh-users/zsh-completions/issues/680">https://github.com/zsh-users/zsh-completions/issues/680</a> I fixed it by doing this:</p>
<pre><code>% compaudit
There are insecure directories:
/usr/local/share/zsh/site-functions
/usr/local/share/zsh
</code></pre>
<p>Then I ran these commands to fix it:</p>
<pre><code>sudo chmod g-w /usr/local/share/zsh/site-functions
sudo chmod g-w /usr/local/share/zsh
</code></pre>


path: macos_find-largest-sqlite.md
topic: macos
title: Finding the largest SQLite files on a Mac
url: https://github.com/simonw/til/blob/main/macos/find-largest-sqlite.md
created: 2020-08-19T08:11:46-07:00
created_utc: 2020-08-19T15:11:46+00:00
updated: 2022-07-26T14:33:15-07:00
updated_utc: 2022-07-26T21:33:15+00:00
shot_hash: 3c5c715b6c2f33d4084647a6af73098f
slug: find-largest-sqlite
body:
This runs using Spotlight so it's really fast:

    mdfind "kMDItemDisplayName == *.sqlite" -0 | xargs -0 stat "-f%z %N" | sort -nr | head -n 20

I have a lot of files in my Dropbox so I excluded those like this:

```
~ % mdfind "kMDItemDisplayName == *.sqlite" -0 | xargs -0 stat "-f%z %N" | sort -nr | grep -v Dropbox | head -n 20
852422656 /Users/simon/Pictures/Photos Library.photoslibrary/database/Photos.sqlite
301924352 /Users/simon/Library/Reminders/Container_v1/Stores/Data-A8BA15F3-C80E-43B3-9E21-F7D1CD42AB6A.sqlite
51343360 /Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite
47349760 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/webappsstore.sqlite
47185920 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/places.sqlite
33779712 /Users/simon/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite
```
Figured out via [this comment](https://news.ycombinator.com/item?id=24179518) by why_only_15 on Hacker News.

## SQLite files in a specific directory

The `-onlyin directory/` option searches just within a specified folder.

Here's how to see all of the SQLite files that have been created by Firefox (which is a lot, because it is used as a disk format for websites that use `localStorage`):

    mdfind "kMDItemDisplayName == *.sqlite" -onlyin ~/Library/Application\ Support/Firefox

html:
<p>This runs using Spotlight so it's really fast:</p>
<pre><code>mdfind "kMDItemDisplayName == *.sqlite" -0 | xargs -0 stat "-f%z %N" | sort -nr | head -n 20
</code></pre>
<p>I have a lot of files in my Dropbox so I excluded those like this:</p>
<pre><code>~ % mdfind "kMDItemDisplayName == *.sqlite" -0 | xargs -0 stat "-f%z %N" | sort -nr | grep -v Dropbox | head -n 20
852422656 /Users/simon/Pictures/Photos Library.photoslibrary/database/Photos.sqlite
301924352 /Users/simon/Library/Reminders/Container_v1/Stores/Data-A8BA15F3-C80E-43B3-9E21-F7D1CD42AB6A.sqlite
51343360 /Users/simon/Pictures/Photos Library.photoslibrary/database/search/psi.sqlite
47349760 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/webappsstore.sqlite
47185920 /Users/simon/Library/Application Support/Firefox/Profiles/ljj897b8.default-release/places.sqlite
33779712 /Users/simon/Library/Group Containers/group.com.apple.notes/NoteStore.sqlite
</code></pre>
<p>Figured out via <a href="https://news.ycombinator.com/item?id=24179518" rel="nofollow">this comment</a> by why_only_15 on Hacker News.</p>
<h2>
<a id="user-content-sqlite-files-in-a-specific-directory" class="anchor" href="#sqlite-files-in-a-specific-directory" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>SQLite files in a specific directory</h2>
<p>The <code>-onlyin directory/</code> option searches just within a specified folder.</p>
<p>Here's how to see all of the SQLite files that have been created by Firefox (which is a lot, because it is used as a disk format for websites that use <code>localStorage</code>):</p>
<pre><code>mdfind "kMDItemDisplayName == *.sqlite" -onlyin ~/Library/Application\ Support/Firefox
</code></pre>


path: macos_zsh-pip-install.md
topic: macos
title: Running pip install -e .[test] in zsh on macOS Catalina
url: https://github.com/simonw/til/blob/main/macos/zsh-pip-install.md
created: 2020-04-21T12:55:07-07:00
created_utc: 2020-04-21T19:55:07+00:00
updated: 2020-04-21T12:55:07-07:00
updated_utc: 2020-04-21T19:55:07+00:00
shot_hash: a0db6b83345270cf36a60840fef37dd0
slug: zsh-pip-install
body:
macOS Catalina uses `zsh` rather than `bash` as the default shell (apparently because Apple don't like GPL 3).

I usually set up my Python projects for development like this:

    datasette % pipenv shell
    Launching subshell in virtual environment…
     . /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/activate                                                                         
    datasette %  . /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/activate
    (datasette) simon@Simons-MacBook-Pro datasette % pip install -e .[test]
    zsh: no matches found: .[test]

In `zsh` the `[` character has special meaning.

Two solutions. The first is to use quotes:

    datasette % pip install -e '.[test]'
    Obtaining file:///Users/simon/Dropbox/Development/datasette
    ...

The second is to prefix it with `noglob`:

    datasette % noglob pip install -e .[test]

html:
<p>macOS Catalina uses <code>zsh</code> rather than <code>bash</code> as the default shell (apparently because Apple don't like GPL 3).</p>
<p>I usually set up my Python projects for development like this:</p>
<pre><code>datasette % pipenv shell
Launching subshell in virtual environment…
 . /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/activate                                                                         
datasette %  . /Users/simon/.local/share/virtualenvs/datasette-AWNrQs95/bin/activate
(datasette) simon@Simons-MacBook-Pro datasette % pip install -e .[test]
zsh: no matches found: .[test]
</code></pre>
<p>In <code>zsh</code> the <code>[</code> character has special meaning.</p>
<p>Two solutions. The first is to use quotes:</p>
<pre><code>datasette % pip install -e '.[test]'
Obtaining file:///Users/simon/Dropbox/Development/datasette
...
</code></pre>
<p>The second is to prefix it with <code>noglob</code>:</p>
<pre><code>datasette % noglob pip install -e .[test]
</code></pre>


path: macos_close-terminal-on-ctrl-d.md
topic: macos
title: Close terminal window on Ctrl+D for macOS
url: https://github.com/simonw/til/blob/main/macos/close-terminal-on-ctrl-d.md
created: 2020-04-21T18:51:26-07:00
created_utc: 2020-04-22T01:51:26+00:00
updated: 2020-04-21T18:51:26-07:00
updated_utc: 2020-04-22T01:51:26+00:00
shot_hash: 38ee1114156f33945737d309d1d712b0
slug: close-terminal-on-ctrl-d
body:
I always forget how to set this up. It's hidden in Preferences -> Profiles -> Basic -> Shell:

![Preferences -> Profiles -> Basic -> Shell](https://raw.githubusercontent.com/simonw/til/master/macos/close-terminal-on-ctrl-d.png)

html:
<p>I always forget how to set this up. It's hidden in Preferences -&gt; Profiles -&gt; Basic -&gt; Shell:</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/master/macos/close-terminal-on-ctrl-d.png" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/master/macos/close-terminal-on-ctrl-d.png" alt="Preferences -&gt; Profiles -&gt; Basic -&gt; Shell" style="max-width:100%;"></a></p>


path: heroku_pg-pull.md
topic: heroku
title: Using heroku pg:pull to restore a backup to a macOS laptop
url: https://github.com/simonw/til/blob/main/heroku/pg-pull.md
created: 2020-07-10T17:16:13-07:00
created_utc: 2020-07-11T00:16:13+00:00
updated: 2020-10-21T16:07:09-07:00
updated_utc: 2020-10-21T23:07:09+00:00
shot_hash: c20e6dd870d76fdfee8dccc3c36d2b2d
slug: pg-pull
body:
Today I worked out how to use the Heroku `pg:pull` command and [Postgres.app](https://postgresapp.com/) to pull a Heroku backup to my laptop.

## Postgres.app

Postgres.app is the easiest way I know to run a PostgreSQL server on macOS: just install using the installer, then click the "Initialize" button in the UI.

## pg:pull

Assuming you have the Heroku CLI installed, the `heroku pg:pull` command can pull a Heroku database backup and load it into a local database.

https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull tells you that you need the following:

    heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi

The app name is your app name (whatever comes before `.herokuapp.com`). 

I found my all-caps environment variable by running the following:

    heroku run env -a simonwillisonblog | grep POSTGRESQL
    Running env on simonwillisonblog... up, run.2963 (Hobby)
    HEROKU_POSTGRESQL_JADE_URL=postgres://....

Then I ran this command:

    heroku pg:pull HEROKU_POSTGRESQL_JADE_URL simonwillisonblog -a simonwillisonblog

This created a local PostgreSQL database called `simonwillisonblog` and imported my latest backup.

When I ran it a second time I had to use `dropdb simonwillisonblog` first to drop the existing local database.

html:
<p>Today I worked out how to use the Heroku <code>pg:pull</code> command and <a href="https://postgresapp.com/" rel="nofollow">Postgres.app</a> to pull a Heroku backup to my laptop.</p>
<h2>
<a id="user-content-postgresapp" class="anchor" href="#postgresapp" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Postgres.app</h2>
<p>Postgres.app is the easiest way I know to run a PostgreSQL server on macOS: just install using the installer, then click the "Initialize" button in the UI.</p>
<h2>
<a id="user-content-pgpull" class="anchor" href="#pgpull" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>pg:pull</h2>
<p>Assuming you have the Heroku CLI installed, the <code>heroku pg:pull</code> command can pull a Heroku database backup and load it into a local database.</p>
<p><a href="https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull" rel="nofollow">https://devcenter.heroku.com/articles/heroku-postgresql#pg-push-and-pg-pull</a> tells you that you need the following:</p>
<pre><code>heroku pg:pull HEROKU_POSTGRESQL_MAGENTA mylocaldb --app sushi
</code></pre>
<p>The app name is your app name (whatever comes before <code>.herokuapp.com</code>).</p>
<p>I found my all-caps environment variable by running the following:</p>
<pre><code>heroku run env -a simonwillisonblog | grep POSTGRESQL
Running env on simonwillisonblog... up, run.2963 (Hobby)
HEROKU_POSTGRESQL_JADE_URL=postgres://....
</code></pre>
<p>Then I ran this command:</p>
<pre><code>heroku pg:pull HEROKU_POSTGRESQL_JADE_URL simonwillisonblog -a simonwillisonblog
</code></pre>
<p>This created a local PostgreSQL database called <code>simonwillisonblog</code> and imported my latest backup.</p>
<p>When I ran it a second time I had to use <code>dropdb simonwillisonblog</code> first to drop the existing local database.</p>


path: heroku_programatic-access-postgresql.md
topic: heroku
title: Programatically accessing Heroku PostgreSQL from GitHub Actions
url: https://github.com/simonw/til/blob/main/heroku/programatic-access-postgresql.md
created: 2020-08-18T15:46:33-07:00
created_utc: 2020-08-18T22:46:33+00:00
updated: 2020-08-18T15:46:33-07:00
updated_utc: 2020-08-18T22:46:33+00:00
shot_hash: 14e4b902d5511a639a6c8d1e91d3dabb
slug: programatic-access-postgresql
body:
My [db-to-sqlite](https://github.com/simonw/db-to-sqlite) tool can connect to a PostgreSQL database, export all of the content and write it to a SQLite database file on disk.

I wanted to use it in a GitHub Action - but that meant I needed code running in the action workflow to be able to access my Heroku PostgreSQL database directly.

It turns out the `DATABASE_URL` environment variable in Heroku has everything you need to connect to that database from elsewhere.

If you have the `heroku` CLU tool installed and authenticated the following one-liner does the job:

    db-to-sqlite $(heroku config:get DATABASE_URL -a simonwillisonblog) simonwillisonblog.db

To configure `heroku` in a GitHub Action you need to set a `HEROKU_API_KEY` environment variable.

You can create an OAuth token on your laptop like this:

    % heroku authorizations:create --scope=read-protected 
    Creating OAuth Authorization... done
    Client:      <none>
    ID:          4dd42e6c-5c89-4389-a9b1-4a5388b88517
    Description: Long-lived user authorization
    Scope:       read-protected
    Token:       xxx copy and paste this bit xxx

Then you can paste the token into a GitHub repository secret called `HEROKU_API_KEY`.

Here's a fragment of my action workflow that creates the SQLite database pulling data from Heroku:

```yaml
    - name: Import Heroku DB into SQLite
      env:
        HEROKU_API_KEY: ${{ secrets.HEROKU_API_KEY }}
      run: |-
        db-to-sqlite $(heroku config:get DATABASE_URL -a simonwillisonblog) simonwillisonblog.db \
          --table auth_permission \
          --table auth_user \
          --table blog_blogmark \
          --table blog_blogmark_tags \
          --table blog_entry \
          --table blog_entry_tags \
          --table blog_quotation \
          --table blog_quotation_tags \
          --table blog_tag \
          --table django_content_type \
          --table redirects_redirect
```

html:
<p>My <a href="https://github.com/simonw/db-to-sqlite">db-to-sqlite</a> tool can connect to a PostgreSQL database, export all of the content and write it to a SQLite database file on disk.</p>
<p>I wanted to use it in a GitHub Action - but that meant I needed code running in the action workflow to be able to access my Heroku PostgreSQL database directly.</p>
<p>It turns out the <code>DATABASE_URL</code> environment variable in Heroku has everything you need to connect to that database from elsewhere.</p>
<p>If you have the <code>heroku</code> CLU tool installed and authenticated the following one-liner does the job:</p>
<pre><code>db-to-sqlite $(heroku config:get DATABASE_URL -a simonwillisonblog) simonwillisonblog.db
</code></pre>
<p>To configure <code>heroku</code> in a GitHub Action you need to set a <code>HEROKU_API_KEY</code> environment variable.</p>
<p>You can create an OAuth token on your laptop like this:</p>
<pre><code>% heroku authorizations:create --scope=read-protected 
Creating OAuth Authorization... done
Client:      &lt;none&gt;
ID:          4dd42e6c-5c89-4389-a9b1-4a5388b88517
Description: Long-lived user authorization
Scope:       read-protected
Token:       xxx copy and paste this bit xxx
</code></pre>
<p>Then you can paste the token into a GitHub repository secret called <code>HEROKU_API_KEY</code>.</p>
<p>Here's a fragment of my action workflow that creates the SQLite database pulling data from Heroku:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Import Heroku DB into SQLite</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">HEROKU_API_KEY</span>: <span class="pl-s">${{ secrets.HEROKU_API_KEY }}</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|-</span>
<span class="pl-s">        db-to-sqlite $(heroku config:get DATABASE_URL -a simonwillisonblog) simonwillisonblog.db \</span>
<span class="pl-s">          --table auth_permission \</span>
<span class="pl-s">          --table auth_user \</span>
<span class="pl-s">          --table blog_blogmark \</span>
<span class="pl-s">          --table blog_blogmark_tags \</span>
<span class="pl-s">          --table blog_entry \</span>
<span class="pl-s">          --table blog_entry_tags \</span>
<span class="pl-s">          --table blog_quotation \</span>
<span class="pl-s">          --table blog_quotation_tags \</span>
<span class="pl-s">          --table blog_tag \</span>
<span class="pl-s">          --table django_content_type \</span>
<span class="pl-s">          --table redirects_redirect</span></pre></div>


path: heroku_pg-upgrade.md
topic: heroku
title: Upgrading a Heroku PostgreSQL database with pg:copy
url: https://github.com/simonw/til/blob/main/heroku/pg-upgrade.md
created: 2020-07-20T11:58:27-07:00
created_utc: 2020-07-20T18:58:27+00:00
updated: 2021-02-26T07:31:57-08:00
updated_utc: 2021-02-26T15:31:57+00:00
shot_hash: 8d9505f3256e62bacfc306cd2ccb207a
slug: pg-upgrade
body:
Figured this out in https://github.com/simonw/simonwillisonblog/issues/132 - I was running PostgreSQL 9.x and I wanted 11.x in order to use the new `search_type="websearch"` option in Django 3.1.

I followed the steps in [Upgrading with pg:copy](https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrading-with-pg-copy) (I don't have any database followers so the `pg:upgrade` route wasn't open to me).

Because I'm the only user who writes to my database I skipped the maintenance mode bit.

## Step one: provision a new DB

    $ heroku addons:create heroku-postgresql:hobby-basic -a simonwillisonblog
    Creating heroku-postgresql:hobby-basic on ⬢ simonwillisonblog... $9/month
    ▸    Release command executing: config vars set by this add-on will not be
    ▸    available until the command succeeds. Use `heroku releases:output` to view
    ▸    the log.
    Database has been created and is available
    ! This database is empty. If upgrading, you can transfer
    ! data from another database with pg:copy
    Created postgresql-animate-15868 as HEROKU_POSTGRESQL_JADE_URL
    Use heroku addons:docs heroku-postgresql to view documentation

## Step 2 skipped - I didn't bother with maintenance mode

## Step 3: copy the data to the new database

I had to wait for it to finish provisioning first:

```
heroku pg:wait -a simonwillisonblog
```
Then I used this command to figure out which `_URL` it had:
```
heroku config -a simonwillisonblog | grep URL
```
Which meant I could run the copy:
```
$ heroku pg:copy DATABASE_URL HEROKU_POSTGRESQL_JADE_URL --app simonwillisonblog
▸    WARNING: Destructive action
▸    This command will remove all data from JADE
▸    Data from DATABASE will then be transferred to JADE
▸    To proceed, type simonwillisonblog or re-run this command with --confirm simonwillisonblog

> simonwillisonblog
Starting copy of DATABASE to JADE... done
Copying... done
```

## Step 4: promote the new database
```
$ heroku pg:promote HEROKU_POSTGRESQL_JADE -a simonwillisonblog
Ensuring an alternate alias for existing DATABASE_URL... HEROKU_POSTGRESQL_OLIVE_URL
Promoting postgresql-animate-15868 to DATABASE_URL on ⬢ simonwillisonblog... done
Checking release phase... pg:promote succeeded. It is safe to ignore the failed Detach DATABASE (@ref:postgresql-contoured-66343) release.
```

## Step 5: deprovision the old database
```
% heroku addons:destroy HEROKU_POSTGRESQL_OLIVE -a simonwillisonblog
 ▸    WARNING: Destructive Action
 ▸    This command will affect the app simonwillisonblog
 ▸    To proceed, type simonwillisonblog or re-run this command with --confirm simonwillisonblog

> simonwillisonblog
Destroying postgresql-contoured-66343 on ⬢ simonwillisonblog... done
```

html:
<p>Figured this out in <a href="https://github.com/simonw/simonwillisonblog/issues/132">https://github.com/simonw/simonwillisonblog/issues/132</a> - I was running PostgreSQL 9.x and I wanted 11.x in order to use the new <code>search_type="websearch"</code> option in Django 3.1.</p>
<p>I followed the steps in <a href="https://devcenter.heroku.com/articles/upgrading-heroku-postgres-databases#upgrading-with-pg-copy" rel="nofollow">Upgrading with pg:copy</a> (I don't have any database followers so the <code>pg:upgrade</code> route wasn't open to me).</p>
<p>Because I'm the only user who writes to my database I skipped the maintenance mode bit.</p>
<h2>
<a id="user-content-step-one-provision-a-new-db" class="anchor" href="#step-one-provision-a-new-db" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Step one: provision a new DB</h2>
<pre><code>$ heroku addons:create heroku-postgresql:hobby-basic -a simonwillisonblog
Creating heroku-postgresql:hobby-basic on ⬢ simonwillisonblog... $9/month
▸    Release command executing: config vars set by this add-on will not be
▸    available until the command succeeds. Use `heroku releases:output` to view
▸    the log.
Database has been created and is available
! This database is empty. If upgrading, you can transfer
! data from another database with pg:copy
Created postgresql-animate-15868 as HEROKU_POSTGRESQL_JADE_URL
Use heroku addons:docs heroku-postgresql to view documentation
</code></pre>
<h2>
<a id="user-content-step-2-skipped---i-didnt-bother-with-maintenance-mode" class="anchor" href="#step-2-skipped---i-didnt-bother-with-maintenance-mode" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Step 2 skipped - I didn't bother with maintenance mode</h2>
<h2>
<a id="user-content-step-3-copy-the-data-to-the-new-database" class="anchor" href="#step-3-copy-the-data-to-the-new-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Step 3: copy the data to the new database</h2>
<p>I had to wait for it to finish provisioning first:</p>
<pre><code>heroku pg:wait -a simonwillisonblog
</code></pre>
<p>Then I used this command to figure out which <code>_URL</code> it had:</p>
<pre><code>heroku config -a simonwillisonblog | grep URL
</code></pre>
<p>Which meant I could run the copy:</p>
<pre><code>$ heroku pg:copy DATABASE_URL HEROKU_POSTGRESQL_JADE_URL --app simonwillisonblog
▸    WARNING: Destructive action
▸    This command will remove all data from JADE
▸    Data from DATABASE will then be transferred to JADE
▸    To proceed, type simonwillisonblog or re-run this command with --confirm simonwillisonblog

&gt; simonwillisonblog
Starting copy of DATABASE to JADE... done
Copying... done
</code></pre>
<h2>
<a id="user-content-step-4-promote-the-new-database" class="anchor" href="#step-4-promote-the-new-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Step 4: promote the new database</h2>
<pre><code>$ heroku pg:promote HEROKU_POSTGRESQL_JADE -a simonwillisonblog
Ensuring an alternate alias for existing DATABASE_URL... HEROKU_POSTGRESQL_OLIVE_URL
Promoting postgresql-animate-15868 to DATABASE_URL on ⬢ simonwillisonblog... done
Checking release phase... pg:promote succeeded. It is safe to ignore the failed Detach DATABASE (@ref:postgresql-contoured-66343) release.
</code></pre>
<h2>
<a id="user-content-step-5-deprovision-the-old-database" class="anchor" href="#step-5-deprovision-the-old-database" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Step 5: deprovision the old database</h2>
<pre><code>% heroku addons:destroy HEROKU_POSTGRESQL_OLIVE -a simonwillisonblog
 ▸    WARNING: Destructive Action
 ▸    This command will affect the app simonwillisonblog
 ▸    To proceed, type simonwillisonblog or re-run this command with --confirm simonwillisonblog

&gt; simonwillisonblog
Destroying postgresql-contoured-66343 on ⬢ simonwillisonblog... done
</code></pre>


path: pytest_mock-httpx.md
topic: pytest
title: How to mock httpx using pytest-mock
url: https://github.com/simonw/til/blob/main/pytest/mock-httpx.md
created: 2020-04-29T10:49:28-07:00
created_utc: 2020-04-29T17:49:28+00:00
updated: 2022-02-01T13:28:21-08:00
updated_utc: 2022-02-01T21:28:21+00:00
shot_hash: 27600afe843c064aa85354d67c4a9c44
slug: mock-httpx
body:
I wrote this test to exercise some [httpx](https://pypi.org/project/httpx/) code today, using [pytest-mock](https://pypi.org/project/pytest-mock/).

The key was to use `mocker.patch.object(cli, "httpx")` which patches the `httpx` module that was imported by the `cli` module.

Here the `mocker` function argument is a fixture that is provided by `pytest-mock`.

```python
from conditional_get import cli
from click.testing import CliRunner


def test_performs_conditional_get(mocker):
    m = mocker.patch.object(cli, "httpx")
    m.get.return_value = mocker.Mock()
    m.get.return_value.status_code = 200
    m.get.return_value.content = b"Hello PNG"
    m.get.return_value.headers = {"etag": "hello-etag"}
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(
            cli.cli, ["https://example.com/file.png", "-o", "file.png"]
        )
        m.get.assert_called_once_with("https://example.com/file.png", headers={})
        assert b"Hello PNG" == open("file.png", "rb").read()
        # Should have also written the ETags file
        assert {"https://example.com/file.png": "hello-etag"} == json.load(
            open("etags.json")
        )
        # Second call should react differently
        m.get.reset_mock()
        m.get.return_value.status_code = 304
        result = runner.invoke(
            cli.cli, ["https://example.com/file.png", "-o", "file.png"]
        )
        m.get.assert_called_once_with(
            "https://example.com/file.png", headers={"If-None-Match": "hello-etag"}
        )
```
https://github.com/simonw/conditional-get/blob/485fab46f01edd99818b829e99765ed9ce0978b5/tests/test_cli.py

## Mocking a JSON response

Here's a mock for a GraphQL POST request that returns JSON:
```python
@pytest.fixture
def mock_graphql_region(mocker):
    m = mocker.patch("datasette_publish_fly.httpx")
    m.post.return_value = mocker.Mock()
    m.post.return_value.status_code = 200
    m.post.return_value.json.return_value = {"data": {"nearestRegion": {"code": "sjc"}}}
```
https://github.com/simonw/datasette-publish-fly/blob/5253220bded001e94561e215d553f352838e7a1c/tests/test_publish_fly.py#L16-L21

## Mocking httpx.stream

I later had to figure out how to mock the following:

```python
with httpx.stream("GET", url, headers=headers) as response:
    ...
    with open(output, "wb") as fp:
        for b in response.iter_bytes():
            fp.write(b)
```
https://stackoverflow.com/a/6112456 helped me figure out the following:
```python
def test_performs_conditional_get(mocker):
    m = mocker.patch.object(cli, "httpx")
    m.stream.return_value.__enter__.return_value = mocker.Mock()
    m.stream.return_value.__enter__.return_value.status_code = 200
    m.stream.return_value.__enter__.return_value.iter_bytes.return_value = [
        b"Hello PNG"
    ]
```
https://github.com/simonw/conditional-get/blob/80454f972d39e2b418572d7938146830fab98fa6/tests/test_cli.py

## Mocking an HTTP error triggered by response.raise_for_status()

The `response.raise_for_status()` raises an exception if an HTTP error (e.g. a 404 or 500) occurred.

Here's how I [mocked that to return an error](https://github.com/simonw/airtable-to-yaml/blob/ebd94b2e29d6f2ec3dc64d161495a759330027e8/tests/test_airtable_to_yaml.py#L43-L56):

```python
def test_airtable_to_yaml_error(mocker):
    m = mocker.patch.object(cli, "httpx")
    m.get.return_value = mocker.Mock()
    m.get.return_value.status_code = 401
    m.get.return_value.raise_for_status.side_effect = httpx.HTTPError(
        "Unauthorized", request=None
    )
    runner = CliRunner()
    with runner.isolated_filesystem():
        result = runner.invoke(
            cli.cli, [".", "appZOGvNJPXCQ205F", "tablename", "-v", "--key", "x"]
        )
        assert result.exit_code == 1
        assert result.stdout == "Error: Unauthorized\n"
```

html:
<p>I wrote this test to exercise some <a href="https://pypi.org/project/httpx/" rel="nofollow">httpx</a> code today, using <a href="https://pypi.org/project/pytest-mock/" rel="nofollow">pytest-mock</a>.</p>
<p>The key was to use <code>mocker.patch.object(cli, "httpx")</code> which patches the <code>httpx</code> module that was imported by the <code>cli</code> module.</p>
<p>Here the <code>mocker</code> function argument is a fixture that is provided by <code>pytest-mock</code>.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">conditional_get</span> <span class="pl-k">import</span> <span class="pl-s1">cli</span>
<span class="pl-k">from</span> <span class="pl-s1">click</span>.<span class="pl-s1">testing</span> <span class="pl-k">import</span> <span class="pl-v">CliRunner</span>


<span class="pl-k">def</span> <span class="pl-en">test_performs_conditional_get</span>(<span class="pl-s1">mocker</span>):
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-s1">patch</span>.<span class="pl-en">object</span>(<span class="pl-s1">cli</span>, <span class="pl-s">"httpx"</span>)
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-v">Mock</span>()
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">=</span> <span class="pl-c1">200</span>
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">content</span> <span class="pl-c1">=</span> <span class="pl-s">b"Hello PNG"</span>
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">headers</span> <span class="pl-c1">=</span> {<span class="pl-s">"etag"</span>: <span class="pl-s">"hello-etag"</span>}
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-k">with</span> <span class="pl-s1">runner</span>.<span class="pl-en">isolated_filesystem</span>():
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(
            <span class="pl-s1">cli</span>.<span class="pl-s1">cli</span>, [<span class="pl-s">"https://example.com/file.png"</span>, <span class="pl-s">"-o"</span>, <span class="pl-s">"file.png"</span>]
        )
        <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-en">assert_called_once_with</span>(<span class="pl-s">"https://example.com/file.png"</span>, <span class="pl-s1">headers</span><span class="pl-c1">=</span>{})
        <span class="pl-k">assert</span> <span class="pl-s">b"Hello PNG"</span> <span class="pl-c1">==</span> <span class="pl-en">open</span>(<span class="pl-s">"file.png"</span>, <span class="pl-s">"rb"</span>).<span class="pl-en">read</span>()
        <span class="pl-c"># Should have also written the ETags file</span>
        <span class="pl-k">assert</span> {<span class="pl-s">"https://example.com/file.png"</span>: <span class="pl-s">"hello-etag"</span>} <span class="pl-c1">==</span> <span class="pl-s1">json</span>.<span class="pl-en">load</span>(
            <span class="pl-en">open</span>(<span class="pl-s">"etags.json"</span>)
        )
        <span class="pl-c"># Second call should react differently</span>
        <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-en">reset_mock</span>()
        <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">=</span> <span class="pl-c1">304</span>
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(
            <span class="pl-s1">cli</span>.<span class="pl-s1">cli</span>, [<span class="pl-s">"https://example.com/file.png"</span>, <span class="pl-s">"-o"</span>, <span class="pl-s">"file.png"</span>]
        )
        <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-en">assert_called_once_with</span>(
            <span class="pl-s">"https://example.com/file.png"</span>, <span class="pl-s1">headers</span><span class="pl-c1">=</span>{<span class="pl-s">"If-None-Match"</span>: <span class="pl-s">"hello-etag"</span>}
        )</pre></div>
<p><a href="https://github.com/simonw/conditional-get/blob/485fab46f01edd99818b829e99765ed9ce0978b5/tests/test_cli.py">https://github.com/simonw/conditional-get/blob/485fab46f01edd99818b829e99765ed9ce0978b5/tests/test_cli.py</a></p>
<h2>
<a id="user-content-mocking-a-json-response" class="anchor" href="#mocking-a-json-response" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Mocking a JSON response</h2>
<p>Here's a mock for a GraphQL POST request that returns JSON:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span></span>
<span class="pl-k">def</span> <span class="pl-en">mock_graphql_region</span>(<span class="pl-s1">mocker</span>):
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-en">patch</span>(<span class="pl-s">"datasette_publish_fly.httpx"</span>)
    <span class="pl-s1">m</span>.<span class="pl-s1">post</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-v">Mock</span>()
    <span class="pl-s1">m</span>.<span class="pl-s1">post</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">=</span> <span class="pl-c1">200</span>
    <span class="pl-s1">m</span>.<span class="pl-s1">post</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">json</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> {<span class="pl-s">"data"</span>: {<span class="pl-s">"nearestRegion"</span>: {<span class="pl-s">"code"</span>: <span class="pl-s">"sjc"</span>}}}</pre></div>
<p><a href="https://github.com/simonw/datasette-publish-fly/blob/5253220bded001e94561e215d553f352838e7a1c/tests/test_publish_fly.py#L16-L21">https://github.com/simonw/datasette-publish-fly/blob/5253220bded001e94561e215d553f352838e7a1c/tests/test_publish_fly.py#L16-L21</a></p>
<h2>
<a id="user-content-mocking-httpxstream" class="anchor" href="#mocking-httpxstream" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Mocking httpx.stream</h2>
<p>I later had to figure out how to mock the following:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">with</span> <span class="pl-s1">httpx</span>.<span class="pl-en">stream</span>(<span class="pl-s">"GET"</span>, <span class="pl-s1">url</span>, <span class="pl-s1">headers</span><span class="pl-c1">=</span><span class="pl-s1">headers</span>) <span class="pl-k">as</span> <span class="pl-s1">response</span>:
    ...
    <span class="pl-k">with</span> <span class="pl-en">open</span>(<span class="pl-s1">output</span>, <span class="pl-s">"wb"</span>) <span class="pl-k">as</span> <span class="pl-s1">fp</span>:
        <span class="pl-k">for</span> <span class="pl-s1">b</span> <span class="pl-c1">in</span> <span class="pl-s1">response</span>.<span class="pl-en">iter_bytes</span>():
            <span class="pl-s1">fp</span>.<span class="pl-en">write</span>(<span class="pl-s1">b</span>)</pre></div>
<p><a href="https://stackoverflow.com/a/6112456" rel="nofollow">https://stackoverflow.com/a/6112456</a> helped me figure out the following:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_performs_conditional_get</span>(<span class="pl-s1">mocker</span>):
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-s1">patch</span>.<span class="pl-en">object</span>(<span class="pl-s1">cli</span>, <span class="pl-s">"httpx"</span>)
    <span class="pl-s1">m</span>.<span class="pl-s1">stream</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">__enter__</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-v">Mock</span>()
    <span class="pl-s1">m</span>.<span class="pl-s1">stream</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">__enter__</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">=</span> <span class="pl-c1">200</span>
    <span class="pl-s1">m</span>.<span class="pl-s1">stream</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">__enter__</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">iter_bytes</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> [
        <span class="pl-s">b"Hello PNG"</span>
    ]</pre></div>
<p><a href="https://github.com/simonw/conditional-get/blob/80454f972d39e2b418572d7938146830fab98fa6/tests/test_cli.py">https://github.com/simonw/conditional-get/blob/80454f972d39e2b418572d7938146830fab98fa6/tests/test_cli.py</a></p>
<h2>
<a id="user-content-mocking-an-http-error-triggered-by-responseraise_for_status" class="anchor" href="#mocking-an-http-error-triggered-by-responseraise_for_status" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Mocking an HTTP error triggered by response.raise_for_status()</h2>
<p>The <code>response.raise_for_status()</code> raises an exception if an HTTP error (e.g. a 404 or 500) occurred.</p>
<p>Here's how I <a href="https://github.com/simonw/airtable-to-yaml/blob/ebd94b2e29d6f2ec3dc64d161495a759330027e8/tests/test_airtable_to_yaml.py#L43-L56">mocked that to return an error</a>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_airtable_to_yaml_error</span>(<span class="pl-s1">mocker</span>):
    <span class="pl-s1">m</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-s1">patch</span>.<span class="pl-en">object</span>(<span class="pl-s1">cli</span>, <span class="pl-s">"httpx"</span>)
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span> <span class="pl-c1">=</span> <span class="pl-s1">mocker</span>.<span class="pl-v">Mock</span>()
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">=</span> <span class="pl-c1">401</span>
    <span class="pl-s1">m</span>.<span class="pl-s1">get</span>.<span class="pl-s1">return_value</span>.<span class="pl-s1">raise_for_status</span>.<span class="pl-s1">side_effect</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-v">HTTPError</span>(
        <span class="pl-s">"Unauthorized"</span>, <span class="pl-s1">request</span><span class="pl-c1">=</span><span class="pl-c1">None</span>
    )
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-k">with</span> <span class="pl-s1">runner</span>.<span class="pl-en">isolated_filesystem</span>():
        <span class="pl-s1">result</span> <span class="pl-c1">=</span> <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(
            <span class="pl-s1">cli</span>.<span class="pl-s1">cli</span>, [<span class="pl-s">"."</span>, <span class="pl-s">"appZOGvNJPXCQ205F"</span>, <span class="pl-s">"tablename"</span>, <span class="pl-s">"-v"</span>, <span class="pl-s">"--key"</span>, <span class="pl-s">"x"</span>]
        )
        <span class="pl-k">assert</span> <span class="pl-s1">result</span>.<span class="pl-s1">exit_code</span> <span class="pl-c1">==</span> <span class="pl-c1">1</span>
        <span class="pl-k">assert</span> <span class="pl-s1">result</span>.<span class="pl-s1">stdout</span> <span class="pl-c1">==</span> <span class="pl-s">"Error: Unauthorized<span class="pl-cce">\n</span>"</span></pre></div>


path: pytest_registering-plugins-in-tests.md
topic: pytest
title: Registering temporary pluggy plugins inside tests
url: https://github.com/simonw/til/blob/main/pytest/registering-plugins-in-tests.md
created: 2020-07-21T09:21:08-07:00
created_utc: 2020-07-21T16:21:08+00:00
updated: 2020-07-31T14:52:06-07:00
updated_utc: 2020-07-31T21:52:06+00:00
shot_hash: f7ddc22ea8a8b13de025d451ac947398
slug: registering-plugins-in-tests
body:
While implementing more finely-grained permissions for `datasette-insert-api` ([issue 8](https://github.com/simonw/datasette-insert-api/issues/8)) I decided I wanted to register a Datasette pluggy plugin for the duration of a single test.

Here's the pattern I figured out for doing that:

```python
from datasette import hookimpl
from datasette.plugins import pm
import pytest


def test_using_test_plugin():
    class TestPlugin:
        __name__ = "TestPlugin"

        @hookimpl
        def permission_allowed(self, datasette, actor, action):
            if action.startswith("insert-api:"):
                return permissions.get(action.replace("insert-api:", ""))

    pm.register(TestPlugin(), name="undo")
    try:
        # Rest of test goes here
    finally:
        pm.unregister(name="undo")
```

Here's [an example](https://github.com/simonw/datasette-insert/blob/7f4c2b3954190d547619d043bbe714481b10ac1e/tests/test_insert_api.py) of a test that uses a pytest fixture to register (and de-register) a plugin:

```python
from datasette import hookimpl
from datasette.app import Datasette
from datasette.plugins import pm
import pytest


@pytest.fixture
def unsafe():
    class UnsafeInsertAll:
        __name__ = "UnsafeInsertAll"

        @hookimpl
        def permission_allowed(self, action):
            if action == "insert:all":
                return True

    pm.register(UnsafeInsertAll(), name="undo")
    yield
    pm.unregister(name="undo")


@pytest.mark.asyncio
async def test_insert_alter(ds, unsafe):
    async with httpx.AsyncClient(app=ds.app()) as client:
        response = await client.post(
            "http://localhost/-/insert/data/dogs?pk=id",
            json=[{"id": 3, "name": "Cleopaws", "age": 5}],
        )
        assert 200 == response.status_code
```

html:
<p>While implementing more finely-grained permissions for <code>datasette-insert-api</code> (<a href="https://github.com/simonw/datasette-insert-api/issues/8">issue 8</a>) I decided I wanted to register a Datasette pluggy plugin for the duration of a single test.</p>
<p>Here's the pattern I figured out for doing that:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">plugins</span> <span class="pl-k">import</span> <span class="pl-s1">pm</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-k">def</span> <span class="pl-en">test_using_test_plugin</span>():
    <span class="pl-k">class</span> <span class="pl-v">TestPlugin</span>:
        <span class="pl-s1">__name__</span> <span class="pl-c1">=</span> <span class="pl-s">"TestPlugin"</span>

        <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
        <span class="pl-k">def</span> <span class="pl-en">permission_allowed</span>(<span class="pl-s1">self</span>, <span class="pl-s1">datasette</span>, <span class="pl-s1">actor</span>, <span class="pl-s1">action</span>):
            <span class="pl-k">if</span> <span class="pl-s1">action</span>.<span class="pl-en">startswith</span>(<span class="pl-s">"insert-api:"</span>):
                <span class="pl-k">return</span> <span class="pl-s1">permissions</span>.<span class="pl-en">get</span>(<span class="pl-s1">action</span>.<span class="pl-en">replace</span>(<span class="pl-s">"insert-api:"</span>, <span class="pl-s">""</span>))

    <span class="pl-s1">pm</span>.<span class="pl-en">register</span>(<span class="pl-v">TestPlugin</span>(), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"undo"</span>)
    <span class="pl-k">try</span>:
        <span class="pl-c"># Rest of test goes here</span>
    <span class="pl-k">finally</span>:
        <span class="pl-s1">pm</span>.<span class="pl-en">unregister</span>(<span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"undo"</span>)</pre></div>
<p>Here's <a href="https://github.com/simonw/datasette-insert/blob/7f4c2b3954190d547619d043bbe714481b10ac1e/tests/test_insert_api.py">an example</a> of a test that uses a pytest fixture to register (and de-register) a plugin:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datasette</span> <span class="pl-k">import</span> <span class="pl-s1">hookimpl</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">app</span> <span class="pl-k">import</span> <span class="pl-v">Datasette</span>
<span class="pl-k">from</span> <span class="pl-s1">datasette</span>.<span class="pl-s1">plugins</span> <span class="pl-k">import</span> <span class="pl-s1">pm</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span></span>
<span class="pl-k">def</span> <span class="pl-en">unsafe</span>():
    <span class="pl-k">class</span> <span class="pl-v">UnsafeInsertAll</span>:
        <span class="pl-s1">__name__</span> <span class="pl-c1">=</span> <span class="pl-s">"UnsafeInsertAll"</span>

        <span class="pl-en">@<span class="pl-s1">hookimpl</span></span>
        <span class="pl-k">def</span> <span class="pl-en">permission_allowed</span>(<span class="pl-s1">self</span>, <span class="pl-s1">action</span>):
            <span class="pl-k">if</span> <span class="pl-s1">action</span> <span class="pl-c1">==</span> <span class="pl-s">"insert:all"</span>:
                <span class="pl-k">return</span> <span class="pl-c1">True</span>

    <span class="pl-s1">pm</span>.<span class="pl-en">register</span>(<span class="pl-v">UnsafeInsertAll</span>(), <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"undo"</span>)
    <span class="pl-k">yield</span>
    <span class="pl-s1">pm</span>.<span class="pl-en">unregister</span>(<span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"undo"</span>)


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">mark</span>.<span class="pl-s1">asyncio</span></span>
<span class="pl-k">async</span> <span class="pl-k">def</span> <span class="pl-en">test_insert_alter</span>(<span class="pl-s1">ds</span>, <span class="pl-s1">unsafe</span>):
    <span class="pl-k">async</span> <span class="pl-k">with</span> <span class="pl-s1">httpx</span>.<span class="pl-v">AsyncClient</span>(<span class="pl-s1">app</span><span class="pl-c1">=</span><span class="pl-s1">ds</span>.<span class="pl-en">app</span>()) <span class="pl-k">as</span> <span class="pl-s1">client</span>:
        <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-s1">client</span>.<span class="pl-en">post</span>(
            <span class="pl-s">"http://localhost/-/insert/data/dogs?pk=id"</span>,
            <span class="pl-s1">json</span><span class="pl-c1">=</span>[{<span class="pl-s">"id"</span>: <span class="pl-c1">3</span>, <span class="pl-s">"name"</span>: <span class="pl-s">"Cleopaws"</span>, <span class="pl-s">"age"</span>: <span class="pl-c1">5</span>}],
        )
        <span class="pl-k">assert</span> <span class="pl-c1">200</span> <span class="pl-c1">==</span> <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span></pre></div>


path: pytest_assert-dictionary-subset.md
topic: pytest
title: Asserting a dictionary is a subset of another dictionary
url: https://github.com/simonw/til/blob/main/pytest/assert-dictionary-subset.md
created: 2020-05-28T08:09:14-07:00
created_utc: 2020-05-28T15:09:14+00:00
updated: 2020-05-28T08:09:14-07:00
updated_utc: 2020-05-28T15:09:14+00:00
shot_hash: a616df055e1f32dc0ebb24f8a75423ac
slug: assert-dictionary-subset
body:
My [lazy approach to writing unit tests](https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/) means that sometimes I want to run an assertion against most (but not all) of a dictionary.

Take for example an API endpoint that returns something like this:

```json
{
    "sqlite_version": "3.27.1",
    "name": "Datasette",
    "columns": ["foo", "bar"]
}
```
I want to efficiently assert against the second two keys, but I don't want to hard-code the SQLite version into my test in case it changes in the future.

Solution:

```python
assert {
    "name": "Datasette",
    "columns": ["foo", "bar"]
}.items() <= api_response.items()
```

The trick here is using `expected.items() <= actual.items()` to assert that one dictionary is a subset of another.

Here's a recent example test that uses this trick: https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/tests/test_plugins.py#L414-L446

html:
<p>My <a href="https://simonwillison.net/2020/Feb/11/cheating-at-unit-tests-pytest-black/" rel="nofollow">lazy approach to writing unit tests</a> means that sometimes I want to run an assertion against most (but not all) of a dictionary.</p>
<p>Take for example an API endpoint that returns something like this:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>sqlite_version<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>3.27.1<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Datasette<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>columns<span class="pl-pds">"</span></span>: [<span class="pl-s"><span class="pl-pds">"</span>foo<span class="pl-pds">"</span></span>, <span class="pl-s"><span class="pl-pds">"</span>bar<span class="pl-pds">"</span></span>]
}</pre></div>
<p>I want to efficiently assert against the second two keys, but I don't want to hard-code the SQLite version into my test in case it changes in the future.</p>
<p>Solution:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">assert</span> {
    <span class="pl-s">"name"</span>: <span class="pl-s">"Datasette"</span>,
    <span class="pl-s">"columns"</span>: [<span class="pl-s">"foo"</span>, <span class="pl-s">"bar"</span>]
}.<span class="pl-en">items</span>() <span class="pl-c1">&lt;=</span> <span class="pl-s1">api_response</span>.<span class="pl-en">items</span>()</pre></div>
<p>The trick here is using <code>expected.items() &lt;= actual.items()</code> to assert that one dictionary is a subset of another.</p>
<p>Here's a recent example test that uses this trick: <a href="https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/tests/test_plugins.py#L414-L446">https://github.com/simonw/datasette/blob/40885ef24e32d91502b6b8bbad1c7376f50f2830/tests/test_plugins.py#L414-L446</a></p>


path: pytest_pytest-code-coverage.md
topic: pytest
title: Code coverage using pytest and codecov.io
url: https://github.com/simonw/til/blob/main/pytest/pytest-code-coverage.md
created: 2020-08-15T11:20:35-07:00
created_utc: 2020-08-15T18:20:35+00:00
updated: 2020-08-15T11:34:18-07:00
updated_utc: 2020-08-15T18:34:18+00:00
shot_hash: c204adc5a1b06a5df221e9be0c1d400c
slug: pytest-code-coverage
body:
I got my [asgi-csrf](https://github.com/simonw/asgi-csrf) Python package up to 100% code coverage. Here's [the pull request](https://github.com/simonw/asgi-csrf/issues/13).

I started by installing and using the [pytest-cov](https://pypi.org/project/pytest-cov/) pytest plugin.

```
pip install pytest-cov
pytest --cov=asgi_csrf
```
This shows the current code coverage percentage for the `asgi_csrf` module in the terminal output:
```
collected 18 items                                                                                                                                                   

test_asgi_csrf.py ..................                                                                                                                           [100%]

---------- coverage: platform darwin, python 3.7.3-final-0 -----------
Name           Stmts   Miss  Cover
----------------------------------
asgi_csrf.py     169     13    92%


========= 18 passed in 0.37s =========
```
To generate an HTML report showing which lines are not covered by tests:
```
pytest --cov=asgi_csrf --cov-report=html
open htmlcov/index.html
```
Here's a hosted copy of that report: https://asgi-csrf-htmlcov-ewca4t9se.vercel.app/asgi_csrf_py.html

## Failing the tests if coverage is below a certain threshold

The `--cov-fail-under=100` option does this:

```
pytest --cov-fail-under=100 --cov asgi_csrf 
======= test session starts =======
platform darwin -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/simon/Dropbox/Development/asgi-csrf
plugins: cov-2.10.1, asyncio-0.14.0
collected 18 items                                                                                                                                                   

test_asgi_csrf.py ..................                                                                                                                           [100%]

---------- coverage: platform darwin, python 3.7.3-final-0 -----------
Name           Stmts   Miss  Cover
----------------------------------
asgi_csrf.py     169     13    92%

FAIL Required test coverage of 100% not reached. Total coverage: 92.31%
```
I added this to my [GitHub test action](https://github.com/simonw/asgi-csrf/blob/83d2b4f6bb034b746fd3f20f57ebdbaeae007a73/.github/workflows/test.yml#L27-L29):
```yaml
    - name: Run tests
      run: |
        pytest --cov-fail-under=100 --cov asgi_csrf
```
## Pushing results to codecov.io

https://codecov.io/ offers free coverage reporting for open source projects. I authorized it against my GitHub account, then enabled it for the `asgi-csrf` project by navigating to https://codecov.io/gh/simonw/asgi-csrf (hacking the URL saves you from having to paginate through all of your repos looking for the right one).

codecov.io gives you a token - set that as a GitHub repository secret as `CODECOV_TOKEN` - then add the following to the test action configuration:
```yaml
    - name: Upload coverage to codecov.io
      run: bash <(curl -s https://codecov.io/bash)
      if: always()
      env:
        CODECOV_TOKEN: ${{ secrets.CODECOV_TOKEN }}
```
This will upload your coverage report (no matter if the previous test step failed or succeeded). codecove.io then reports back to pull requests and maintains a dashboard for your project.

codecov.io doesn't detect if you use a `main` or `master` branch so I had to switch the default branch in the settings at https://codecov.io/gh/simonw/asgi-csrf/settings

I also added their badge to my project README, using the following Markdown:
```
[![codecov](https://codecov.io/gh/simonw/asgi-csrf/branch/main/graph/badge.svg)](https://codecov.io/gh/simonw/asgi-csrf)
```

html:
<p>I got my <a href="https://github.com/simonw/asgi-csrf">asgi-csrf</a> Python package up to 100% code coverage. Here's <a href="https://github.com/simonw/asgi-csrf/issues/13">the pull request</a>.</p>
<p>I started by installing and using the <a href="https://pypi.org/project/pytest-cov/" rel="nofollow">pytest-cov</a> pytest plugin.</p>
<pre><code>pip install pytest-cov
pytest --cov=asgi_csrf
</code></pre>
<p>This shows the current code coverage percentage for the <code>asgi_csrf</code> module in the terminal output:</p>
<pre><code>collected 18 items                                                                                                                                                   

test_asgi_csrf.py ..................                                                                                                                           [100%]

---------- coverage: platform darwin, python 3.7.3-final-0 -----------
Name           Stmts   Miss  Cover
----------------------------------
asgi_csrf.py     169     13    92%


========= 18 passed in 0.37s =========
</code></pre>
<p>To generate an HTML report showing which lines are not covered by tests:</p>
<pre><code>pytest --cov=asgi_csrf --cov-report=html
open htmlcov/index.html
</code></pre>
<p>Here's a hosted copy of that report: <a href="https://asgi-csrf-htmlcov-ewca4t9se.vercel.app/asgi_csrf_py.html" rel="nofollow">https://asgi-csrf-htmlcov-ewca4t9se.vercel.app/asgi_csrf_py.html</a></p>
<h2>
<a id="user-content-failing-the-tests-if-coverage-is-below-a-certain-threshold" class="anchor" href="#failing-the-tests-if-coverage-is-below-a-certain-threshold" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Failing the tests if coverage is below a certain threshold</h2>
<p>The <code>--cov-fail-under=100</code> option does this:</p>
<pre><code>pytest --cov-fail-under=100 --cov asgi_csrf 
======= test session starts =======
platform darwin -- Python 3.7.3, pytest-6.0.1, py-1.9.0, pluggy-0.13.1
rootdir: /Users/simon/Dropbox/Development/asgi-csrf
plugins: cov-2.10.1, asyncio-0.14.0
collected 18 items                                                                                                                                                   

test_asgi_csrf.py ..................                                                                                                                           [100%]

---------- coverage: platform darwin, python 3.7.3-final-0 -----------
Name           Stmts   Miss  Cover
----------------------------------
asgi_csrf.py     169     13    92%

FAIL Required test coverage of 100% not reached. Total coverage: 92.31%
</code></pre>
<p>I added this to my <a href="https://github.com/simonw/asgi-csrf/blob/83d2b4f6bb034b746fd3f20f57ebdbaeae007a73/.github/workflows/test.yml#L27-L29">GitHub test action</a>:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Run tests</span>
      <span class="pl-ent">run</span>: <span class="pl-s">|</span>
<span class="pl-s">        pytest --cov-fail-under=100 --cov asgi_csrf</span></pre></div>
<h2>
<a id="user-content-pushing-results-to-codecovio" class="anchor" href="#pushing-results-to-codecovio" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Pushing results to codecov.io</h2>
<p><a href="https://codecov.io/" rel="nofollow">https://codecov.io/</a> offers free coverage reporting for open source projects. I authorized it against my GitHub account, then enabled it for the <code>asgi-csrf</code> project by navigating to <a href="https://codecov.io/gh/simonw/asgi-csrf" rel="nofollow">https://codecov.io/gh/simonw/asgi-csrf</a> (hacking the URL saves you from having to paginate through all of your repos looking for the right one).</p>
<p>codecov.io gives you a token - set that as a GitHub repository secret as <code>CODECOV_TOKEN</code> - then add the following to the test action configuration:</p>
<div class="highlight highlight-source-yaml"><pre>    - <span class="pl-ent">name</span>: <span class="pl-s">Upload coverage to codecov.io</span>
      <span class="pl-ent">run</span>: <span class="pl-s">bash &lt;(curl -s https://codecov.io/bash)</span>
      <span class="pl-ent">if</span>: <span class="pl-s">always()</span>
      <span class="pl-ent">env</span>:
        <span class="pl-ent">CODECOV_TOKEN</span>: <span class="pl-s">${{ secrets.CODECOV_TOKEN }}</span></pre></div>
<p>This will upload your coverage report (no matter if the previous test step failed or succeeded). codecove.io then reports back to pull requests and maintains a dashboard for your project.</p>
<p>codecov.io doesn't detect if you use a <code>main</code> or <code>master</code> branch so I had to switch the default branch in the settings at <a href="https://codecov.io/gh/simonw/asgi-csrf/settings" rel="nofollow">https://codecov.io/gh/simonw/asgi-csrf/settings</a></p>
<p>I also added their badge to my project README, using the following Markdown:</p>
<pre><code>[![codecov](https://codecov.io/gh/simonw/asgi-csrf/branch/main/graph/badge.svg)](https://codecov.io/gh/simonw/asgi-csrf)
</code></pre>


path: pytest_subprocess-server.md
topic: pytest
title: Start a server in a subprocess during a pytest session
url: https://github.com/simonw/til/blob/main/pytest/subprocess-server.md
created: 2020-08-31T19:06:36-07:00
created_utc: 2020-09-01T02:06:36+00:00
updated: 2022-07-22T18:20:37-07:00
updated_utc: 2022-07-23T01:20:37+00:00
shot_hash: a5ecf996dc1a8af46fecd371d7a1063e
slug: subprocess-server
body:
I wanted to start an actual server process, run it for the duration of my pytest session and shut it down at the end.

Here's the recipe I came up with. This fixture lives in `conftest.py`:

```python
import pytest
import sqlite_utils
import subprocess

@pytest.fixture(scope="session")
def ds_server(tmp_path_factory):
    db_directory = tmp_path_factory.mktemp("dbs")
    db_path = db_directory / "test.db"
    db = sqlite_utils.Database(db_path)
    insert_test_data(db)
    ds_proc = subprocess.Popen(
        [
            "datasette",
            str(db_path),
            "-p",
            "8041"
        ],
        stdout=subprocess.PIPE,
        stderr=subprocess.STDOUT,
    )
    # Give the server time to start
    time.sleep(2)
    # Check it started successfully
    assert not ds_proc.poll(), ds_proc.stdout.read().decode("utf-8")
    yield ds_proc
    # Shut it down at the end of the pytest session
    ds_proc.terminate()
```
A test looks like this:
```python
import httpx

def test_server_starts(ds_server):
    response = httpx.get("http://127.0.0.1:8041/")
    assert response.status_code == 200
```

## Alternative recipe for serving static files

While [adding tests to Datasette Lite](https://github.com/simonw/datasette-lite/issues/35) I found myself needing to run a localhost server that served static files directly.

I completely forgot about this TIL, and instead took inspiration [from pytest-simplehttpserver](https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/pytest_plugin.py#L17-L28) - coming up with this pattern:

```python
from subprocess import Popen, PIPE
import pathlib
import pytest
import time
from http.client import HTTPConnection

root = pathlib.Path(__file__).parent.parent.absolute()


@pytest.fixture(scope="module")
def static_server():
    process = Popen(
        ["python", "-m", "http.server", "8123", "--directory", root], stdout=PIPE
    )
    retries = 5
    while retries > 0:
        conn = HTTPConnection("localhost:8123")
        try:
            conn.request("HEAD", "/")
            response = conn.getresponse()
            if response is not None:
                yield process
                break
        except ConnectionRefusedError:
            time.sleep(1)
            retries -= 1

    if not retries:
        raise RuntimeError("Failed to start http server")
    else:
        process.terminate()
        process.wait()
```
Again, including `static_server` as a fixture is enough to ensure requests to `http://localhost:8123/` will be served by that temporary server.

I like how this version polls for a successful HEAD request (a trick inspired by `pytest-simplehttpserver`) rather than just sleeping.

html:
<p>I wanted to start an actual server process, run it for the duration of my pytest session and shut it down at the end.</p>
<p>Here's the recipe I came up with. This fixture lives in <code>conftest.py</code>:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest</span>
<span class="pl-k">import</span> <span class="pl-s1">sqlite_utils</span>
<span class="pl-k">import</span> <span class="pl-s1">subprocess</span>

<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-en">fixture</span>(<span class="pl-s1">scope</span><span class="pl-c1">=</span><span class="pl-s">"session"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">ds_server</span>(<span class="pl-s1">tmp_path_factory</span>):
    <span class="pl-s1">db_directory</span> <span class="pl-c1">=</span> <span class="pl-s1">tmp_path_factory</span>.<span class="pl-en">mktemp</span>(<span class="pl-s">"dbs"</span>)
    <span class="pl-s1">db_path</span> <span class="pl-c1">=</span> <span class="pl-s1">db_directory</span> <span class="pl-c1">/</span> <span class="pl-s">"test.db"</span>
    <span class="pl-s1">db</span> <span class="pl-c1">=</span> <span class="pl-s1">sqlite_utils</span>.<span class="pl-v">Database</span>(<span class="pl-s1">db_path</span>)
    <span class="pl-en">insert_test_data</span>(<span class="pl-s1">db</span>)
    <span class="pl-s1">ds_proc</span> <span class="pl-c1">=</span> <span class="pl-s1">subprocess</span>.<span class="pl-v">Popen</span>(
        [
            <span class="pl-s">"datasette"</span>,
            <span class="pl-en">str</span>(<span class="pl-s1">db_path</span>),
            <span class="pl-s">"-p"</span>,
            <span class="pl-s">"8041"</span>
        ],
        <span class="pl-s1">stdout</span><span class="pl-c1">=</span><span class="pl-s1">subprocess</span>.<span class="pl-v">PIPE</span>,
        <span class="pl-s1">stderr</span><span class="pl-c1">=</span><span class="pl-s1">subprocess</span>.<span class="pl-v">STDOUT</span>,
    )
    <span class="pl-c"># Give the server time to start</span>
    <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">2</span>)
    <span class="pl-c"># Check it started successfully</span>
    <span class="pl-k">assert</span> <span class="pl-c1">not</span> <span class="pl-s1">ds_proc</span>.<span class="pl-en">poll</span>(), <span class="pl-s1">ds_proc</span>.<span class="pl-s1">stdout</span>.<span class="pl-en">read</span>().<span class="pl-en">decode</span>(<span class="pl-s">"utf-8"</span>)
    <span class="pl-k">yield</span> <span class="pl-s1">ds_proc</span>
    <span class="pl-c"># Shut it down at the end of the pytest session</span>
    <span class="pl-s1">ds_proc</span>.<span class="pl-en">terminate</span>()</pre></div>
<p>A test looks like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">httpx</span>

<span class="pl-k">def</span> <span class="pl-en">test_server_starts</span>(<span class="pl-s1">ds_server</span>):
    <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">get</span>(<span class="pl-s">"http://127.0.0.1:8041/"</span>)
    <span class="pl-k">assert</span> <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">==</span> <span class="pl-c1">200</span></pre></div>
<h2>
<a id="user-content-alternative-recipe-for-serving-static-files" class="anchor" href="#alternative-recipe-for-serving-static-files" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Alternative recipe for serving static files</h2>
<p>While <a href="https://github.com/simonw/datasette-lite/issues/35">adding tests to Datasette Lite</a> I found myself needing to run a localhost server that served static files directly.</p>
<p>I completely forgot about this TIL, and instead took inspiration <a href="https://github.com/ppmdo/pytest-simplehttpserver/blob/a82ad31912121c074ff1a76c4628a1c42c32b41b/src/pytest_simplehttpserver/pytest_plugin.py#L17-L28">from pytest-simplehttpserver</a> - coming up with this pattern:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">subprocess</span> <span class="pl-k">import</span> <span class="pl-v">Popen</span>, <span class="pl-v">PIPE</span>
<span class="pl-k">import</span> <span class="pl-s1">pathlib</span>
<span class="pl-k">import</span> <span class="pl-s1">pytest</span>
<span class="pl-k">import</span> <span class="pl-s1">time</span>
<span class="pl-k">from</span> <span class="pl-s1">http</span>.<span class="pl-s1">client</span> <span class="pl-k">import</span> <span class="pl-v">HTTPConnection</span>

<span class="pl-s1">root</span> <span class="pl-c1">=</span> <span class="pl-s1">pathlib</span>.<span class="pl-v">Path</span>(<span class="pl-s1">__file__</span>).<span class="pl-s1">parent</span>.<span class="pl-s1">parent</span>.<span class="pl-en">absolute</span>()


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-en">fixture</span>(<span class="pl-s1">scope</span><span class="pl-c1">=</span><span class="pl-s">"module"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">static_server</span>():
    <span class="pl-s1">process</span> <span class="pl-c1">=</span> <span class="pl-v">Popen</span>(
        [<span class="pl-s">"python"</span>, <span class="pl-s">"-m"</span>, <span class="pl-s">"http.server"</span>, <span class="pl-s">"8123"</span>, <span class="pl-s">"--directory"</span>, <span class="pl-s1">root</span>], <span class="pl-s1">stdout</span><span class="pl-c1">=</span><span class="pl-v">PIPE</span>
    )
    <span class="pl-s1">retries</span> <span class="pl-c1">=</span> <span class="pl-c1">5</span>
    <span class="pl-k">while</span> <span class="pl-s1">retries</span> <span class="pl-c1">&gt;</span> <span class="pl-c1">0</span>:
        <span class="pl-s1">conn</span> <span class="pl-c1">=</span> <span class="pl-v">HTTPConnection</span>(<span class="pl-s">"localhost:8123"</span>)
        <span class="pl-k">try</span>:
            <span class="pl-s1">conn</span>.<span class="pl-en">request</span>(<span class="pl-s">"HEAD"</span>, <span class="pl-s">"/"</span>)
            <span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">conn</span>.<span class="pl-en">getresponse</span>()
            <span class="pl-k">if</span> <span class="pl-s1">response</span> <span class="pl-c1">is</span> <span class="pl-c1">not</span> <span class="pl-c1">None</span>:
                <span class="pl-k">yield</span> <span class="pl-s1">process</span>
                <span class="pl-k">break</span>
        <span class="pl-k">except</span> <span class="pl-v">ConnectionRefusedError</span>:
            <span class="pl-s1">time</span>.<span class="pl-en">sleep</span>(<span class="pl-c1">1</span>)
            <span class="pl-s1">retries</span> <span class="pl-c1">-=</span> <span class="pl-c1">1</span>

    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-s1">retries</span>:
        <span class="pl-k">raise</span> <span class="pl-v">RuntimeError</span>(<span class="pl-s">"Failed to start http server"</span>)
    <span class="pl-k">else</span>:
        <span class="pl-s1">process</span>.<span class="pl-en">terminate</span>()
        <span class="pl-s1">process</span>.<span class="pl-en">wait</span>()</pre></div>
<p>Again, including <code>static_server</code> as a fixture is enough to ensure requests to <code>http://localhost:8123/</code> will be served by that temporary server.</p>
<p>I like how this version polls for a successful HEAD request (a trick inspired by <code>pytest-simplehttpserver</code>) rather than just sleeping.</p>


path: pytest_session-scoped-tmp.md
topic: pytest
title: Session-scoped temporary directories in pytest
url: https://github.com/simonw/til/blob/main/pytest/session-scoped-tmp.md
created: 2020-04-26T20:47:07-07:00
created_utc: 2020-04-27T03:47:07+00:00
updated: 2020-04-26T20:47:07-07:00
updated_utc: 2020-04-27T03:47:07+00:00
shot_hash: 89134f29475508a4bbea0574f4ee0340
slug: session-scoped-tmp
body:
I habitually use the `tmpdir` fixture in pytest to get a temporary directory that will be cleaned up after each test, but that doesn't work with `scope="session"` - which can be used to ensure an expensive fixture is run only once per test session and the generated content is used for multiple tests.

To get a temporary directory that works with `scope="session"`, use the `tmp_path_factory` built-in pytest fixture like this:

```python
import pytest


@pytest.fixture(scope="session")
def template_dir(tmp_path_factory):
    template_dir = tmp_path_factory.mktemp("page-templates")
    pages_dir = template_dir / "pages"
    pages_dir.mkdir()
    (pages_dir / "about.html").write_text("ABOUT!", "utf-8")
    (pages_dir / "request.html").write_text("request", "utf-8")
    return template_dir


def test_about(template_dir):
    assert "ABOUT!" == (template_dir / "pages" / "about.html").read_text()


def test_request(template_dir):
    assert "request" == (template_dir / "pages" / "request.html").read_text()
```

Example: https://github.com/simonw/datasette/blob/1b7b66c465e44025ec73421bd69752e42f108321/tests/test_custom_pages.py#L16-L45

html:
<p>I habitually use the <code>tmpdir</code> fixture in pytest to get a temporary directory that will be cleaned up after each test, but that doesn't work with <code>scope="session"</code> - which can be used to ensure an expensive fixture is run only once per test session and the generated content is used for multiple tests.</p>
<p>To get a temporary directory that works with <code>scope="session"</code>, use the <code>tmp_path_factory</code> built-in pytest fixture like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">pytest</span>


<span class="pl-en">@<span class="pl-s1">pytest</span>.<span class="pl-s1">fixture</span>(<span class="pl-s1">scope</span><span class="pl-c1">=</span><span class="pl-s">"session"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">template_dir</span>(<span class="pl-s1">tmp_path_factory</span>):
    <span class="pl-s1">template_dir</span> <span class="pl-c1">=</span> <span class="pl-s1">tmp_path_factory</span>.<span class="pl-en">mktemp</span>(<span class="pl-s">"page-templates"</span>)
    <span class="pl-s1">pages_dir</span> <span class="pl-c1">=</span> <span class="pl-s1">template_dir</span> <span class="pl-c1">/</span> <span class="pl-s">"pages"</span>
    <span class="pl-s1">pages_dir</span>.<span class="pl-en">mkdir</span>()
    (<span class="pl-s1">pages_dir</span> <span class="pl-c1">/</span> <span class="pl-s">"about.html"</span>).<span class="pl-en">write_text</span>(<span class="pl-s">"ABOUT!"</span>, <span class="pl-s">"utf-8"</span>)
    (<span class="pl-s1">pages_dir</span> <span class="pl-c1">/</span> <span class="pl-s">"request.html"</span>).<span class="pl-en">write_text</span>(<span class="pl-s">"request"</span>, <span class="pl-s">"utf-8"</span>)
    <span class="pl-k">return</span> <span class="pl-s1">template_dir</span>


<span class="pl-k">def</span> <span class="pl-en">test_about</span>(<span class="pl-s1">template_dir</span>):
    <span class="pl-k">assert</span> <span class="pl-s">"ABOUT!"</span> <span class="pl-c1">==</span> (<span class="pl-s1">template_dir</span> <span class="pl-c1">/</span> <span class="pl-s">"pages"</span> <span class="pl-c1">/</span> <span class="pl-s">"about.html"</span>).<span class="pl-en">read_text</span>()


<span class="pl-k">def</span> <span class="pl-en">test_request</span>(<span class="pl-s1">template_dir</span>):
    <span class="pl-k">assert</span> <span class="pl-s">"request"</span> <span class="pl-c1">==</span> (<span class="pl-s1">template_dir</span> <span class="pl-c1">/</span> <span class="pl-s">"pages"</span> <span class="pl-c1">/</span> <span class="pl-s">"request.html"</span>).<span class="pl-en">read_text</span>()</pre></div>
<p>Example: <a href="https://github.com/simonw/datasette/blob/1b7b66c465e44025ec73421bd69752e42f108321/tests/test_custom_pages.py#L16-L45">https://github.com/simonw/datasette/blob/1b7b66c465e44025ec73421bd69752e42f108321/tests/test_custom_pages.py#L16-L45</a></p>


path: python_convert-to-utc-without-pytz.md
topic: python
title: Convert a datetime object to UTC without using pytz
url: https://github.com/simonw/til/blob/main/python/convert-to-utc-without-pytz.md
created: 2020-04-19T09:02:23-07:00
created_utc: 2020-04-19T16:02:23+00:00
updated: 2020-04-19T09:02:23-07:00
updated_utc: 2020-04-19T16:02:23+00:00
shot_hash: 1193605e5392b17d869cfccc224bea57
slug: convert-to-utc-without-pytz
body:
I wanted to convert a datetime object (from GitPython) to UTC without adding the `pytz` dependency.

```python
from datetime import timezone
import git


repo = git.Repo(".", odbt=git.GitDB)
commit = list(repo.iter_commits(ref))[0]
dt = commit.committed_datetime
# This was 2020-04-19T07:55:08-07:00
dt_in_utc = dt.astimezone(timezone.utc)

# Now use .isoformat() to convert to a string
print(dt_in_utc.isoformat())
# Came out as 2020-04-19T14:55:08+00:00
```

html:
<p>I wanted to convert a datetime object (from GitPython) to UTC without adding the <code>pytz</code> dependency.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">from</span> <span class="pl-s1">datetime</span> <span class="pl-k">import</span> <span class="pl-s1">timezone</span>
<span class="pl-k">import</span> <span class="pl-s1">git</span>


<span class="pl-s1">repo</span> <span class="pl-c1">=</span> <span class="pl-s1">git</span>.<span class="pl-v">Repo</span>(<span class="pl-s">"."</span>, <span class="pl-s1">odbt</span><span class="pl-c1">=</span><span class="pl-s1">git</span>.<span class="pl-v">GitDB</span>)
<span class="pl-s1">commit</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">repo</span>.<span class="pl-en">iter_commits</span>(<span class="pl-s1">ref</span>))[<span class="pl-c1">0</span>]
<span class="pl-s1">dt</span> <span class="pl-c1">=</span> <span class="pl-s1">commit</span>.<span class="pl-s1">committed_datetime</span>
<span class="pl-c"># This was 2020-04-19T07:55:08-07:00</span>
<span class="pl-s1">dt_in_utc</span> <span class="pl-c1">=</span> <span class="pl-s1">dt</span>.<span class="pl-en">astimezone</span>(<span class="pl-s1">timezone</span>.<span class="pl-s1">utc</span>)

<span class="pl-c"># Now use .isoformat() to convert to a string</span>
<span class="pl-en">print</span>(<span class="pl-s1">dt_in_utc</span>.<span class="pl-en">isoformat</span>())
<span class="pl-c"># Came out as 2020-04-19T14:55:08+00:00</span></pre></div>


path: python_call-pip-programatically.md
topic: python
title: How to call pip programatically from Python
url: https://github.com/simonw/til/blob/main/python/call-pip-programatically.md
created: 2020-08-11T17:07:25-07:00
created_utc: 2020-08-12T00:07:25+00:00
updated: 2020-08-11T17:07:25-07:00
updated_utc: 2020-08-12T00:07:25+00:00
shot_hash: 9f6ddb3d03aec145bafbabc482340a8c
slug: call-pip-programatically
body:
I needed this for the `datasette install` and `datasette uninstall` commands, see [issue #925](https://github.com/simonw/datasette/issues/925).

My initial attempt at this resulted in weird testing errors ([#928](https://github.com/simonw/datasette/issues/928)) - while investigating them I stumbled across [this comment](https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47) in the `pip` source code:

```
# Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.

# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.

# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.

# The following example shows how to use runpy to invoke pip in that
# case:
#
#     sys.argv = ["pip", your, args, here]
#     runpy.run_module("pip", run_name="__main__")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.
```
So I did that! Here's the working version of my `datasette install` command:
```python
 @cli.command() 
 @click.argument("packages", nargs=-1, required=True) 
 def install(packages): 
     "Install Python packages - e.g. Datasette plugins - into the same environment as Datasette" 
     sys.argv = ["pip", "install"] + list(packages) 
     run_module("pip", run_name="__main__") 
```
And here's how I wrote [a unit test](https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/tests/test_cli.py#L114-L124) for it:
```python
@mock.patch("datasette.cli.run_module")
def test_install(run_module):
    runner = CliRunner()
    runner.invoke(cli, ["install", "datasette-mock-plugin", "datasette-mock-plugin2"])
    run_module.assert_called_once_with("pip", run_name="__main__")
    assert sys.argv == [
        "pip",
        "install",
        "datasette-mock-plugin",
        "datasette-mock-plugin2",
    ]
```

html:
<p>I needed this for the <code>datasette install</code> and <code>datasette uninstall</code> commands, see <a href="https://github.com/simonw/datasette/issues/925">issue #925</a>.</p>
<p>My initial attempt at this resulted in weird testing errors (<a href="https://github.com/simonw/datasette/issues/928">#928</a>) - while investigating them I stumbled across <a href="https://github.com/pypa/pip/blob/e060970d51c5946beac8447eb95585d83019582d/src/pip/_internal/cli/main.py#L23-L47">this comment</a> in the <code>pip</code> source code:</p>
<pre><code># Do not import and use main() directly! Using it directly is actively
# discouraged by pip's maintainers. The name, location and behavior of
# this function is subject to change, so calling it directly is not
# portable across different pip versions.

# In addition, running pip in-process is unsupported and unsafe. This is
# elaborated in detail at
# https://pip.pypa.io/en/stable/user_guide/#using-pip-from-your-program.
# That document also provides suggestions that should work for nearly
# all users that are considering importing and using main() directly.

# However, we know that certain users will still want to invoke pip
# in-process. If you understand and accept the implications of using pip
# in an unsupported manner, the best approach is to use runpy to avoid
# depending on the exact location of this entry point.

# The following example shows how to use runpy to invoke pip in that
# case:
#
#     sys.argv = ["pip", your, args, here]
#     runpy.run_module("pip", run_name="__main__")
#
# Note that this will exit the process after running, unlike a direct
# call to main. As it is not safe to do any processing after calling
# main, this should not be an issue in practice.
</code></pre>
<p>So I did that! Here's the working version of my <code>datasette install</code> command:</p>
<div class="highlight highlight-source-python"><pre> <span class="pl-en">@<span class="pl-s1">cli</span>.<span class="pl-s1">command</span>()</span> 
 <span class="pl-en">@<span class="pl-s1">click</span>.<span class="pl-s1">argument</span>(<span class="pl-s">"packages"</span>, <span class="pl-s1">nargs</span><span class="pl-c1">=</span><span class="pl-c1">-</span><span class="pl-c1">1</span>, <span class="pl-s1">required</span><span class="pl-c1">=</span><span class="pl-c1">True</span>)</span> 
 <span class="pl-k">def</span> <span class="pl-en">install</span>(<span class="pl-s1">packages</span>): 
     <span class="pl-s">"Install Python packages - e.g. Datasette plugins - into the same environment as Datasette"</span> 
     <span class="pl-s1">sys</span>.<span class="pl-s1">argv</span> <span class="pl-c1">=</span> [<span class="pl-s">"pip"</span>, <span class="pl-s">"install"</span>] <span class="pl-c1">+</span> <span class="pl-en">list</span>(<span class="pl-s1">packages</span>) 
     <span class="pl-en">run_module</span>(<span class="pl-s">"pip"</span>, <span class="pl-s1">run_name</span><span class="pl-c1">=</span><span class="pl-s">"__main__"</span>) </pre></div>
<p>And here's how I wrote <a href="https://github.com/simonw/datasette/blob/afdeda8216d4d3027f87583ccdbef17ad85022ef/tests/test_cli.py#L114-L124">a unit test</a> for it:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">@<span class="pl-s1">mock</span>.<span class="pl-s1">patch</span>(<span class="pl-s">"datasette.cli.run_module"</span>)</span>
<span class="pl-k">def</span> <span class="pl-en">test_install</span>(<span class="pl-s1">run_module</span>):
    <span class="pl-s1">runner</span> <span class="pl-c1">=</span> <span class="pl-v">CliRunner</span>()
    <span class="pl-s1">runner</span>.<span class="pl-en">invoke</span>(<span class="pl-s1">cli</span>, [<span class="pl-s">"install"</span>, <span class="pl-s">"datasette-mock-plugin"</span>, <span class="pl-s">"datasette-mock-plugin2"</span>])
    <span class="pl-s1">run_module</span>.<span class="pl-en">assert_called_once_with</span>(<span class="pl-s">"pip"</span>, <span class="pl-s1">run_name</span><span class="pl-c1">=</span><span class="pl-s">"__main__"</span>)
    <span class="pl-k">assert</span> <span class="pl-s1">sys</span>.<span class="pl-s1">argv</span> <span class="pl-c1">==</span> [
        <span class="pl-s">"pip"</span>,
        <span class="pl-s">"install"</span>,
        <span class="pl-s">"datasette-mock-plugin"</span>,
        <span class="pl-s">"datasette-mock-plugin2"</span>,
    ]</pre></div>


path: python_json-floating-point.md
topic: python
title: Outputting JSON with reduced floating point precision
url: https://github.com/simonw/til/blob/main/python/json-floating-point.md
created: 2020-08-21T21:19:26-07:00
created_utc: 2020-08-22T04:19:26+00:00
updated: 2020-08-21T21:19:26-07:00
updated_utc: 2020-08-22T04:19:26+00:00
shot_hash: aee3a12c03e9e3d849ba3bc1d8efd203
slug: json-floating-point
body:
[datasette-leaflet-geojson](github.com/simonw/datasette-leaflet-geojson) outputs GeoJSON geometries in HTML pages in a way that can be picked up by JavaScript and used to plot a Leaflet map.

These geometries often look something like this:

```
{
  "type":"MultiPolygon",
  "coordinates":[[[[-122.4457678900319,37.77292891669105],[-122.441075063058,37.77352490695095]...
```
[Decimal_degrees: Precision on Wikipedia](https://en.wikipedia.org/wiki/Decimal_degrees#Precision) says that `0.00001` should be accurate to within around a meter.

Shortening these floating point representations can shave 100KB+ off an HTML page with a lot of GeoJSON shapes on it!

There's [a lengthy Stack Overflow](https://stackoverflow.com/questions/1447287/format-floats-with-standard-json-module) about this but it's difficult to follow because ways of doing this changed between Python 2 and Python 3. Here's what worked for me:
```python

def round_floats(o):
    if isinstance(o, float):
        return round(o, 5)
    if isinstance(o, dict):
        return {k: round_floats(v) for k, v in o.items()}
    if isinstance(o, (list, tuple)):
        return [round_floats(x) for x in o]
    return o

data = json.dumps(round_floats(data))
```
See [issue 11](https://github.com/simonw/datasette-leaflet-geojson/issues/11) for details.

html:
<p><a href="github.com/simonw/datasette-leaflet-geojson">datasette-leaflet-geojson</a> outputs GeoJSON geometries in HTML pages in a way that can be picked up by JavaScript and used to plot a Leaflet map.</p>
<p>These geometries often look something like this:</p>
<pre><code>{
  "type":"MultiPolygon",
  "coordinates":[[[[-122.4457678900319,37.77292891669105],[-122.441075063058,37.77352490695095]...
</code></pre>
<p><a href="https://en.wikipedia.org/wiki/Decimal_degrees#Precision" rel="nofollow">Decimal_degrees: Precision on Wikipedia</a> says that <code>0.00001</code> should be accurate to within around a meter.</p>
<p>Shortening these floating point representations can shave 100KB+ off an HTML page with a lot of GeoJSON shapes on it!</p>
<p>There's <a href="https://stackoverflow.com/questions/1447287/format-floats-with-standard-json-module" rel="nofollow">a lengthy Stack Overflow</a> about this but it's difficult to follow because ways of doing this changed between Python 2 and Python 3. Here's what worked for me:</p>
<div class="highlight highlight-source-python"><pre>
<span class="pl-k">def</span> <span class="pl-en">round_floats</span>(<span class="pl-s1">o</span>):
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">o</span>, <span class="pl-s1">float</span>):
        <span class="pl-k">return</span> <span class="pl-en">round</span>(<span class="pl-s1">o</span>, <span class="pl-c1">5</span>)
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">o</span>, <span class="pl-s1">dict</span>):
        <span class="pl-k">return</span> {<span class="pl-s1">k</span>: <span class="pl-en">round_floats</span>(<span class="pl-s1">v</span>) <span class="pl-k">for</span> <span class="pl-s1">k</span>, <span class="pl-s1">v</span> <span class="pl-c1">in</span> <span class="pl-s1">o</span>.<span class="pl-en">items</span>()}
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">o</span>, (<span class="pl-s1">list</span>, <span class="pl-s1">tuple</span>)):
        <span class="pl-k">return</span> [<span class="pl-en">round_floats</span>(<span class="pl-s1">x</span>) <span class="pl-k">for</span> <span class="pl-s1">x</span> <span class="pl-c1">in</span> <span class="pl-s1">o</span>]
    <span class="pl-k">return</span> <span class="pl-s1">o</span>

<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">json</span>.<span class="pl-en">dumps</span>(<span class="pl-en">round_floats</span>(<span class="pl-s1">data</span>))</pre></div>
<p>See <a href="https://github.com/simonw/datasette-leaflet-geojson/issues/11">issue 11</a> for details.</p>


path: python_build-official-docs.md
topic: python
title: Build the official Python documentation locally
url: https://github.com/simonw/til/blob/main/python/build-official-docs.md
created: 2020-05-08T09:16:57-07:00
created_utc: 2020-05-08T16:16:57+00:00
updated: 2022-07-31T19:10:54-07:00
updated_utc: 2022-08-01T02:10:54+00:00
shot_hash: 4f58459fd908669c38836e80878466e7
slug: build-official-docs
body:
First, checkout the cpython repo:

    $ git clone git@github.com:python/cpython

cd into the `Doc` directory:

    $ cd cpython/Doc

To get a virtual environment with Sphinx and other required tools, run `make venv` like this:

    $ make venv
    python3 -m venv ./venv
    ./venv/bin/python3 -m pip install -U pip setuptools
    ...
    Installing collected packages: snowballstemmer, sphinxcontrib-jsmath, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, pyparsing, six, packaging, sphinxcontrib-applehelp, sphinxcontrib-qthelp, alabaster, docutils, MarkupSafe, Jinja2, pytz, babel, imagesize, chardet, certifi, urllib3, idna, requests, Pygments, sphinxcontrib-htmlhelp, Sphinx, blurb, python-docs-theme
        Running setup.py install for python-docs-theme ... done
    Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.6.1 Sphinx-2.2.0 alabaster-0.7.12 babel-2.8.0 blurb-1.0.7 certifi-2020.4.5.1 chardet-3.0.4 docutils-0.16 idna-2.9 imagesize-1.2.0 packaging-20.3 pyparsing-2.4.7 python-docs-theme-2020.1 pytz-2020.1 requests-2.23.0 six-1.14.0 snowballstemmer-2.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.25.9
    The venv has been created in the ./venv directory

Now running `make html` will build the docs:

    $ make html

Run `open build/html/index.html` to view them in a browser.

html:
<p>First, checkout the cpython repo:</p>
<pre><code>$ git clone git@github.com:python/cpython
</code></pre>
<p>cd into the <code>Doc</code> directory:</p>
<pre><code>$ cd cpython/Doc
</code></pre>
<p>To get a virtual environment with Sphinx and other required tools, run <code>make venv</code> like this:</p>
<pre><code>$ make venv
python3 -m venv ./venv
./venv/bin/python3 -m pip install -U pip setuptools
...
Installing collected packages: snowballstemmer, sphinxcontrib-jsmath, sphinxcontrib-devhelp, sphinxcontrib-serializinghtml, pyparsing, six, packaging, sphinxcontrib-applehelp, sphinxcontrib-qthelp, alabaster, docutils, MarkupSafe, Jinja2, pytz, babel, imagesize, chardet, certifi, urllib3, idna, requests, Pygments, sphinxcontrib-htmlhelp, Sphinx, blurb, python-docs-theme
    Running setup.py install for python-docs-theme ... done
Successfully installed Jinja2-2.11.2 MarkupSafe-1.1.1 Pygments-2.6.1 Sphinx-2.2.0 alabaster-0.7.12 babel-2.8.0 blurb-1.0.7 certifi-2020.4.5.1 chardet-3.0.4 docutils-0.16 idna-2.9 imagesize-1.2.0 packaging-20.3 pyparsing-2.4.7 python-docs-theme-2020.1 pytz-2020.1 requests-2.23.0 six-1.14.0 snowballstemmer-2.0.0 sphinxcontrib-applehelp-1.0.2 sphinxcontrib-devhelp-1.0.2 sphinxcontrib-htmlhelp-1.0.3 sphinxcontrib-jsmath-1.0.1 sphinxcontrib-qthelp-1.0.3 sphinxcontrib-serializinghtml-1.1.4 urllib3-1.25.9
The venv has been created in the ./venv directory
</code></pre>
<p>Now running <code>make html</code> will build the docs:</p>
<pre><code>$ make html
</code></pre>
<p>Run <code>open build/html/index.html</code> to view them in a browser.</p>


path: python_introspect-function-parameters.md
topic: python
title: Introspecting Python function parameters
url: https://github.com/simonw/til/blob/main/python/introspect-function-parameters.md
created: 2020-05-27T15:43:05-07:00
created_utc: 2020-05-27T22:43:05+00:00
updated: 2020-05-27T15:43:05-07:00
updated_utc: 2020-05-27T22:43:05+00:00
shot_hash: dbec643394b9e7ff1b9bf89db54ee409
slug: introspect-function-parameters
body:
For https://github.com/simonw/datasette/issues/581 I want to be able to inspect a Python function to determine which named parameters it accepts and send only those arguments.

Python 3.3 added [an inspect.signature() function](https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object) that can be used for this.

Here's a function I wrote to take advantage of that and solve my problem:

```python
def call_with_supported_arguments(fn, **kwargs):
    parameters = inspect.signature(fn).parameters.keys()
    call_with = []
    for parameter in parameters:
        if parameter not in kwargs:
            raise TypeError("{} requires parameters {}".format(fn, tuple(parameters)))
        call_with.append(kwargs[parameter])
    return fn(*call_with)
```

And here's an illustrative unit test:

```python
def test_call_with_supported_arguments():
    def foo(a, b):
        return "{}+{}".format(a, b)

    assert "1+2" == utils.call_with_supported_arguments(foo, a=1, b=2)
    assert "1+2" == utils.call_with_supported_arguments(foo, a=1, b=2, c=3)

    with pytest.raises(TypeError):
        utils.call_with_supported_arguments(foo, a=1)
```

html:
<p>For <a href="https://github.com/simonw/datasette/issues/581">https://github.com/simonw/datasette/issues/581</a> I want to be able to inspect a Python function to determine which named parameters it accepts and send only those arguments.</p>
<p>Python 3.3 added <a href="https://docs.python.org/3/library/inspect.html#introspecting-callables-with-the-signature-object" rel="nofollow">an inspect.signature() function</a> that can be used for this.</p>
<p>Here's a function I wrote to take advantage of that and solve my problem:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">call_with_supported_arguments</span>(<span class="pl-s1">fn</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>):
    <span class="pl-s1">parameters</span> <span class="pl-c1">=</span> <span class="pl-s1">inspect</span>.<span class="pl-en">signature</span>(<span class="pl-s1">fn</span>).<span class="pl-s1">parameters</span>.<span class="pl-en">keys</span>()
    <span class="pl-s1">call_with</span> <span class="pl-c1">=</span> []
    <span class="pl-k">for</span> <span class="pl-s1">parameter</span> <span class="pl-c1">in</span> <span class="pl-s1">parameters</span>:
        <span class="pl-k">if</span> <span class="pl-s1">parameter</span> <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">kwargs</span>:
            <span class="pl-k">raise</span> <span class="pl-v">TypeError</span>(<span class="pl-s">"{} requires parameters {}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">fn</span>, <span class="pl-en">tuple</span>(<span class="pl-s1">parameters</span>)))
        <span class="pl-s1">call_with</span>.<span class="pl-en">append</span>(<span class="pl-s1">kwargs</span>[<span class="pl-s1">parameter</span>])
    <span class="pl-k">return</span> <span class="pl-en">fn</span>(<span class="pl-c1">*</span><span class="pl-s1">call_with</span>)</pre></div>
<p>And here's an illustrative unit test:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">test_call_with_supported_arguments</span>():
    <span class="pl-k">def</span> <span class="pl-en">foo</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>):
        <span class="pl-k">return</span> <span class="pl-s">"{}+{}"</span>.<span class="pl-en">format</span>(<span class="pl-s1">a</span>, <span class="pl-s1">b</span>)

    <span class="pl-k">assert</span> <span class="pl-s">"1+2"</span> <span class="pl-c1">==</span> <span class="pl-s1">utils</span>.<span class="pl-en">call_with_supported_arguments</span>(<span class="pl-s1">foo</span>, <span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-c1">2</span>)
    <span class="pl-k">assert</span> <span class="pl-s">"1+2"</span> <span class="pl-c1">==</span> <span class="pl-s1">utils</span>.<span class="pl-en">call_with_supported_arguments</span>(<span class="pl-s1">foo</span>, <span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">1</span>, <span class="pl-s1">b</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">c</span><span class="pl-c1">=</span><span class="pl-c1">3</span>)

    <span class="pl-k">with</span> <span class="pl-s1">pytest</span>.<span class="pl-en">raises</span>(<span class="pl-v">TypeError</span>):
        <span class="pl-s1">utils</span>.<span class="pl-en">call_with_supported_arguments</span>(<span class="pl-s1">foo</span>, <span class="pl-s1">a</span><span class="pl-c1">=</span><span class="pl-c1">1</span>)</pre></div>


path: python_macos-catalina-sort-of-ships-with-python3.md
topic: python
title: macOS Catalina sort-of includes Python 3
url: https://github.com/simonw/til/blob/main/python/macos-catalina-sort-of-ships-with-python3.md
created: 2020-04-21T11:19:54-07:00
created_utc: 2020-04-21T18:19:54+00:00
updated: 2020-04-21T11:22:01-07:00
updated_utc: 2020-04-21T18:22:01+00:00
shot_hash: b9f53e00783bd9b1e3581b24a1ff1c68
slug: macos-catalina-sort-of-ships-with-python3
body:
Once you have installed the "command line tools" for Catalina using the following terminal command:

    xcode-select --install

`python3` will run Python 3.7.3:

    % which python3
    /usr/bin/python3
    % python3
    Python 3.7.3 (default, Apr  7 2020, 14:06:47) 
    [Clang 11.0.3 (clang-1103.0.32.59)] on darwin
    Type "help", "copyright", "credits" or "license" for more information.
    >>> 

`pip` isn't there as a standalone tool on the command-line, but `python3 -mpip install x` will work:

```
python3 -mpip install github-to-sqlite
Collecting github-to-sqlite
  Downloading https://files.pythonhosted.org/packages/18/d2/8a622d3fdbe161517df6941be257d0ed4590108593c3aff9a35578fa381e/github_to_sqlite-1.1-py3-none-any.whl
...
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/3.7'
Consider using the `--user` option or check the permissions.
```
I don't like running `pip` as root, so I try the `--user` option:

```
% python3 -mpip install github-to-sqlite --user
Collecting github-to-sqlite
  Using cached https://files.pythonhosted.org/packages/18/d2/8a622d3fdbe161517df6941be257d0ed4590108593c3aff9a35578fa381e/github_to_sqlite-1.1-py3-none-any.whl
Requirement already satisfied: requests in ./Library/Python/3.7/lib/python/site-packages (from github-to-sqlite) (2.23.0)
Collecting sqlite-utils>=2.7 (from github-to-sqlite)
  Using cached https://files.pythonhosted.org/packages/26/af/91237b71616a3c63bbb921ba8f0039fc784d0d33e3f908c727c1d39c757a/sqlite_utils-2.7-py3-none-any.whl
Requirement already satisfied: chardet<4,>=3.0.2 in ./Library/Python/3.7/lib/python/site-packages (from requests->github-to-sqlite) (3.0.4)
Requirement already satisfied: idna<3,>=2.5 in ./Library/Python/3.7/lib/python/site-packages (from requests->github-to-sqlite) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,<1.26,>=1.21.1 in ./Library/Python/3.7/lib/python/site-packages (from requests->github-to-sqlite) (1.25.9)
Requirement already satisfied: certifi>=2017.4.17 in ./Library/Python/3.7/lib/python/site-packages (from requests->github-to-sqlite) (2020.4.5.1)
Requirement already satisfied: tabulate in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils>=2.7->github-to-sqlite) (0.8.7)
Requirement already satisfied: click in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils>=2.7->github-to-sqlite) (7.1.1)
Requirement already satisfied: click-default-group in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils>=2.7->github-to-sqlite) (1.2.2)
pocket-to-sqlite 0.2 has requirement sqlite-utils~=2.4.4, but you'll have sqlite-utils 2.7 which is incompatible.
Installing collected packages: sqlite-utils, github-to-sqlite
  Found existing installation: sqlite-utils 2.4.4
    Uninstalling sqlite-utils-2.4.4:
      Successfully uninstalled sqlite-utils-2.4.4
  The script sqlite-utils is installed in '/Users/simon/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script github-to-sqlite is installed in '/Users/simon/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed github-to-sqlite-1.1 sqlite-utils-2.7
```
As per the warnings, this isn't on my path: I can run it like this:

```
/Users/simon/Library/Python/3.7/bin/github-to-sqlite
```
Or I can add that directory to my path.

Which I didn't do, because I decided to use homebrew's Python 3.8 instead, with `pipenv` and `pipx`:
```
% brew install pipenv
# ...
% brew install pipx
Updating Homebrew...
==> Downloading https://homebrew.bintray.com/bottles/pipx-0.15.1.3_1.catalina.bottle.tar.gz
######################################################################## 100.0%
==> Pouring pipx-0.15.1.3_1.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/pipx/0.15.1.3_1: 92 files, 566.8KB
```
Now I'll try using it to install a command-line tool ([github-to-sqlite](https://github.com/dogsheep/github-to-sqlite)):
```
% pipx install github-to-sqlite
  installed package github-to-sqlite 1.1, Python 3.8.2
  These apps are now globally available
    - github-to-sqlite
⚠️  Note: '/Users/simon/.local/bin' is not on your PATH environment variable. These apps will not be globally accessible until your PATH is updated. Run `pipx ensurepath` to automatically add it, or manually modify your PATH in your shell's config file (i.e. ~/.bashrc).
done! ✨ 🌟 ✨
```
In this case I WILL add `~.local/bin` to my PATH:
```
% pipx ensurepath
```
This put the following in my `.zshrc`:
```
% cat .zshrc      

# Created by `userpath` on 2020-04-21 18:18:34
export PATH="$PATH:/Users/simon/.local/bin"
```

html:
<p>Once you have installed the "command line tools" for Catalina using the following terminal command:</p>
<pre><code>xcode-select --install
</code></pre>
<p><code>python3</code> will run Python 3.7.3:</p>
<pre><code>% which python3
/usr/bin/python3
% python3
Python 3.7.3 (default, Apr  7 2020, 14:06:47) 
[Clang 11.0.3 (clang-1103.0.32.59)] on darwin
Type "help", "copyright", "credits" or "license" for more information.
&gt;&gt;&gt; 
</code></pre>
<p><code>pip</code> isn't there as a standalone tool on the command-line, but <code>python3 -mpip install x</code> will work:</p>
<pre><code>python3 -mpip install github-to-sqlite
Collecting github-to-sqlite
  Downloading https://files.pythonhosted.org/packages/18/d2/8a622d3fdbe161517df6941be257d0ed4590108593c3aff9a35578fa381e/github_to_sqlite-1.1-py3-none-any.whl
...
Could not install packages due to an EnvironmentError: [Errno 13] Permission denied: '/Library/Python/3.7'
Consider using the `--user` option or check the permissions.
</code></pre>
<p>I don't like running <code>pip</code> as root, so I try the <code>--user</code> option:</p>
<pre><code>% python3 -mpip install github-to-sqlite --user
Collecting github-to-sqlite
  Using cached https://files.pythonhosted.org/packages/18/d2/8a622d3fdbe161517df6941be257d0ed4590108593c3aff9a35578fa381e/github_to_sqlite-1.1-py3-none-any.whl
Requirement already satisfied: requests in ./Library/Python/3.7/lib/python/site-packages (from github-to-sqlite) (2.23.0)
Collecting sqlite-utils&gt;=2.7 (from github-to-sqlite)
  Using cached https://files.pythonhosted.org/packages/26/af/91237b71616a3c63bbb921ba8f0039fc784d0d33e3f908c727c1d39c757a/sqlite_utils-2.7-py3-none-any.whl
Requirement already satisfied: chardet&lt;4,&gt;=3.0.2 in ./Library/Python/3.7/lib/python/site-packages (from requests-&gt;github-to-sqlite) (3.0.4)
Requirement already satisfied: idna&lt;3,&gt;=2.5 in ./Library/Python/3.7/lib/python/site-packages (from requests-&gt;github-to-sqlite) (2.9)
Requirement already satisfied: urllib3!=1.25.0,!=1.25.1,&lt;1.26,&gt;=1.21.1 in ./Library/Python/3.7/lib/python/site-packages (from requests-&gt;github-to-sqlite) (1.25.9)
Requirement already satisfied: certifi&gt;=2017.4.17 in ./Library/Python/3.7/lib/python/site-packages (from requests-&gt;github-to-sqlite) (2020.4.5.1)
Requirement already satisfied: tabulate in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils&gt;=2.7-&gt;github-to-sqlite) (0.8.7)
Requirement already satisfied: click in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils&gt;=2.7-&gt;github-to-sqlite) (7.1.1)
Requirement already satisfied: click-default-group in ./Library/Python/3.7/lib/python/site-packages (from sqlite-utils&gt;=2.7-&gt;github-to-sqlite) (1.2.2)
pocket-to-sqlite 0.2 has requirement sqlite-utils~=2.4.4, but you'll have sqlite-utils 2.7 which is incompatible.
Installing collected packages: sqlite-utils, github-to-sqlite
  Found existing installation: sqlite-utils 2.4.4
    Uninstalling sqlite-utils-2.4.4:
      Successfully uninstalled sqlite-utils-2.4.4
  The script sqlite-utils is installed in '/Users/simon/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
  The script github-to-sqlite is installed in '/Users/simon/Library/Python/3.7/bin' which is not on PATH.
  Consider adding this directory to PATH or, if you prefer to suppress this warning, use --no-warn-script-location.
Successfully installed github-to-sqlite-1.1 sqlite-utils-2.7
</code></pre>
<p>As per the warnings, this isn't on my path: I can run it like this:</p>
<pre><code>/Users/simon/Library/Python/3.7/bin/github-to-sqlite
</code></pre>
<p>Or I can add that directory to my path.</p>
<p>Which I didn't do, because I decided to use homebrew's Python 3.8 instead, with <code>pipenv</code> and <code>pipx</code>:</p>
<pre><code>% brew install pipenv
# ...
% brew install pipx
Updating Homebrew...
==&gt; Downloading https://homebrew.bintray.com/bottles/pipx-0.15.1.3_1.catalina.bottle.tar.gz
######################################################################## 100.0%
==&gt; Pouring pipx-0.15.1.3_1.catalina.bottle.tar.gz
🍺  /usr/local/Cellar/pipx/0.15.1.3_1: 92 files, 566.8KB
</code></pre>
<p>Now I'll try using it to install a command-line tool (<a href="https://github.com/dogsheep/github-to-sqlite">github-to-sqlite</a>):</p>
<pre><code>% pipx install github-to-sqlite
  installed package github-to-sqlite 1.1, Python 3.8.2
  These apps are now globally available
    - github-to-sqlite
⚠️  Note: '/Users/simon/.local/bin' is not on your PATH environment variable. These apps will not be globally accessible until your PATH is updated. Run `pipx ensurepath` to automatically add it, or manually modify your PATH in your shell's config file (i.e. ~/.bashrc).
done! ✨ 🌟 ✨
</code></pre>
<p>In this case I WILL add <code>~.local/bin</code> to my PATH:</p>
<pre><code>% pipx ensurepath
</code></pre>
<p>This put the following in my <code>.zshrc</code>:</p>
<pre><code>% cat .zshrc      

# Created by `userpath` on 2020-04-21 18:18:34
export PATH="$PATH:/Users/simon/.local/bin"
</code></pre>


path: python_password-hashing-with-pbkdf2.md
topic: python
title: Password hashing in Python with pbkdf2
url: https://github.com/simonw/til/blob/main/python/password-hashing-with-pbkdf2.md
created: 2020-07-13T20:53:56-07:00
created_utc: 2020-07-14T03:53:56+00:00
updated: 2020-07-13T20:54:40-07:00
updated_utc: 2020-07-14T03:54:40+00:00
shot_hash: d85626b1c24f179b540fbb8001e93af8
slug: password-hashing-with-pbkdf2
body:
I was researching password hashing for [datasette-auth-passwords](https://github.com/simonw/datasette-auth-passwords). I wanted very secure defaults that would work using the Python standard library without any extra dependencies.

I ended up following [Django's example](https://github.com/django/django/blob/136ec9b62bd0b105f281218d7cad54b7db7a4bab/django/contrib/auth/hashers.py#L247) and implementing `pbkdf2_sha256` with 260,000 iterations. Here's my version of the way Django does this - I chose to use the same hashed password format (`algorithm$iterations$salt$hash`) in case I wanted to change the hashing algorithm used in the future.

```python
import base64
import hashlib
import secrets

ALGORITHM = "pbkdf2_sha256"


def hash_password(password, salt=None, iterations=260000):
    if salt is None:
        salt = secrets.token_hex(16)
    assert salt and isinstance(salt, str) and "$" not in salt
    assert isinstance(password, str)
    pw_hash = hashlib.pbkdf2_hmac(
        "sha256", password.encode("utf-8"), salt.encode("utf-8"), iterations
    )
    b64_hash = base64.b64encode(pw_hash).decode("ascii").strip()
    return "{}${}${}${}".format(ALGORITHM, iterations, salt, b64_hash)


def verify_password(password, password_hash):
    if (password_hash or "").count("$") != 3:
        return False
    algorithm, iterations, salt, b64_hash = password_hash.split("$", 3)
    iterations = int(iterations)
    assert algorithm == ALGORITHM
    compare_hash = hash_password(password, salt, iterations)
    return secrets.compare_digest(password_hash, compare_hash)
```
Code here: https://github.com/simonw/datasette-auth-passwords/blob/ac538faf510d8ea7cf8064b9567d09c6a69e079a/datasette_auth_passwords/utils.py

html:
<p>I was researching password hashing for <a href="https://github.com/simonw/datasette-auth-passwords">datasette-auth-passwords</a>. I wanted very secure defaults that would work using the Python standard library without any extra dependencies.</p>
<p>I ended up following <a href="https://github.com/django/django/blob/136ec9b62bd0b105f281218d7cad54b7db7a4bab/django/contrib/auth/hashers.py#L247">Django's example</a> and implementing <code>pbkdf2_sha256</code> with 260,000 iterations. Here's my version of the way Django does this - I chose to use the same hashed password format (<code>algorithm$iterations$salt$hash</code>) in case I wanted to change the hashing algorithm used in the future.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">base64</span>
<span class="pl-k">import</span> <span class="pl-s1">hashlib</span>
<span class="pl-k">import</span> <span class="pl-s1">secrets</span>

<span class="pl-v">ALGORITHM</span> <span class="pl-c1">=</span> <span class="pl-s">"pbkdf2_sha256"</span>


<span class="pl-k">def</span> <span class="pl-en">hash_password</span>(<span class="pl-s1">password</span>, <span class="pl-s1">salt</span><span class="pl-c1">=</span><span class="pl-c1">None</span>, <span class="pl-s1">iterations</span><span class="pl-c1">=</span><span class="pl-c1">260000</span>):
    <span class="pl-k">if</span> <span class="pl-s1">salt</span> <span class="pl-c1">is</span> <span class="pl-c1">None</span>:
        <span class="pl-s1">salt</span> <span class="pl-c1">=</span> <span class="pl-s1">secrets</span>.<span class="pl-en">token_hex</span>(<span class="pl-c1">16</span>)
    <span class="pl-k">assert</span> <span class="pl-s1">salt</span> <span class="pl-c1">and</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">salt</span>, <span class="pl-s1">str</span>) <span class="pl-c1">and</span> <span class="pl-s">"$"</span> <span class="pl-c1">not</span> <span class="pl-c1">in</span> <span class="pl-s1">salt</span>
    <span class="pl-k">assert</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">password</span>, <span class="pl-s1">str</span>)
    <span class="pl-s1">pw_hash</span> <span class="pl-c1">=</span> <span class="pl-s1">hashlib</span>.<span class="pl-en">pbkdf2_hmac</span>(
        <span class="pl-s">"sha256"</span>, <span class="pl-s1">password</span>.<span class="pl-en">encode</span>(<span class="pl-s">"utf-8"</span>), <span class="pl-s1">salt</span>.<span class="pl-en">encode</span>(<span class="pl-s">"utf-8"</span>), <span class="pl-s1">iterations</span>
    )
    <span class="pl-s1">b64_hash</span> <span class="pl-c1">=</span> <span class="pl-s1">base64</span>.<span class="pl-en">b64encode</span>(<span class="pl-s1">pw_hash</span>).<span class="pl-en">decode</span>(<span class="pl-s">"ascii"</span>).<span class="pl-en">strip</span>()
    <span class="pl-k">return</span> <span class="pl-s">"{}${}${}${}"</span>.<span class="pl-en">format</span>(<span class="pl-v">ALGORITHM</span>, <span class="pl-s1">iterations</span>, <span class="pl-s1">salt</span>, <span class="pl-s1">b64_hash</span>)


<span class="pl-k">def</span> <span class="pl-en">verify_password</span>(<span class="pl-s1">password</span>, <span class="pl-s1">password_hash</span>):
    <span class="pl-k">if</span> (<span class="pl-s1">password_hash</span> <span class="pl-c1">or</span> <span class="pl-s">""</span>).<span class="pl-en">count</span>(<span class="pl-s">"$"</span>) <span class="pl-c1">!=</span> <span class="pl-c1">3</span>:
        <span class="pl-k">return</span> <span class="pl-c1">False</span>
    <span class="pl-s1">algorithm</span>, <span class="pl-s1">iterations</span>, <span class="pl-s1">salt</span>, <span class="pl-s1">b64_hash</span> <span class="pl-c1">=</span> <span class="pl-s1">password_hash</span>.<span class="pl-en">split</span>(<span class="pl-s">"$"</span>, <span class="pl-c1">3</span>)
    <span class="pl-s1">iterations</span> <span class="pl-c1">=</span> <span class="pl-en">int</span>(<span class="pl-s1">iterations</span>)
    <span class="pl-k">assert</span> <span class="pl-s1">algorithm</span> <span class="pl-c1">==</span> <span class="pl-v">ALGORITHM</span>
    <span class="pl-s1">compare_hash</span> <span class="pl-c1">=</span> <span class="pl-en">hash_password</span>(<span class="pl-s1">password</span>, <span class="pl-s1">salt</span>, <span class="pl-s1">iterations</span>)
    <span class="pl-k">return</span> <span class="pl-s1">secrets</span>.<span class="pl-en">compare_digest</span>(<span class="pl-s1">password_hash</span>, <span class="pl-s1">compare_hash</span>)</pre></div>
<p>Code here: <a href="https://github.com/simonw/datasette-auth-passwords/blob/ac538faf510d8ea7cf8064b9567d09c6a69e079a/datasette_auth_passwords/utils.py">https://github.com/simonw/datasette-auth-passwords/blob/ac538faf510d8ea7cf8064b9567d09c6a69e079a/datasette_auth_passwords/utils.py</a></p>


path: python_installing-upgrading-plugins-with-pipx.md
topic: python
title: Installing and upgrading Datasette plugins with pipx
url: https://github.com/simonw/til/blob/main/python/installing-upgrading-plugins-with-pipx.md
created: 2020-05-04T12:14:25-07:00
created_utc: 2020-05-04T19:14:25+00:00
updated: 2020-12-29T13:55:23-08:00
updated_utc: 2020-12-29T21:55:23+00:00
shot_hash: 555dcee3dede51206c0cbf017ab41a8d
slug: installing-upgrading-plugins-with-pipx
body:
If you installed `datasette` using `pipx install datasette` you can install additional plugins with `pipx inject` like so:

    $ datasette plugins
    []

    $ pipx inject datasette datasette-json-html
      injected package datasette-json-html into venv datasette
    done! ✨ 🌟 ✨

    $ datasette plugins
    [
        {
            "name": "datasette-json-html",
            "static": false,
            "templates": false,
            "version": "0.6"
        }
    ]

Thanks [Matthew Somerville](https://twitter.com/dracos/status/1257351655907090437).

I then had to figure out how to upgrade them. Thanks to https://github.com/pipxproject/pipx/issues/79 I figured out the following recipe using `pipx runpip datasette install -U name-of-plugin`: 

    % datasette plugins
    [
        {
            "name": "datasette-vega",
            "static": true,
            "templates": false,
            "version": "0.6"
        }
    ]

    $ pipx runpip datasette install -U datasette-vega
    Collecting datasette-vega
    Downloading datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
        |████████████████████████████████| 1.8 MB 2.0 MB/s
    ...
    Installing collected packages: datasette-vega
    Attempting uninstall: datasette-vega
        Found existing installation: datasette-vega 0.6
        Uninstalling datasette-vega-0.6:
        Successfully uninstalled datasette-vega-0.6
    Successfully installed datasette-vega-0.6.2

    $ datasette plugins
    [
        {
            "name": "datasette-vega",
            "static": true,
            "templates": false,
            "version": "0.6.2"
        }
    ]

I added all of this to the Datasette docs here: https://docs.datasette.io/en/latest/installation.html#using-pipx
(see https://github.com/simonw/datasette/issues/756).

html:
<p>If you installed <code>datasette</code> using <code>pipx install datasette</code> you can install additional plugins with <code>pipx inject</code> like so:</p>
<pre><code>$ datasette plugins
[]

$ pipx inject datasette datasette-json-html
  injected package datasette-json-html into venv datasette
done! ✨ 🌟 ✨

$ datasette plugins
[
    {
        "name": "datasette-json-html",
        "static": false,
        "templates": false,
        "version": "0.6"
    }
]
</code></pre>
<p>Thanks <a href="https://twitter.com/dracos/status/1257351655907090437" rel="nofollow">Matthew Somerville</a>.</p>
<p>I then had to figure out how to upgrade them. Thanks to <a href="https://github.com/pipxproject/pipx/issues/79">https://github.com/pipxproject/pipx/issues/79</a> I figured out the following recipe using <code>pipx runpip datasette install -U name-of-plugin</code>:</p>
<pre><code>% datasette plugins
[
    {
        "name": "datasette-vega",
        "static": true,
        "templates": false,
        "version": "0.6"
    }
]

$ pipx runpip datasette install -U datasette-vega
Collecting datasette-vega
Downloading datasette_vega-0.6.2-py3-none-any.whl (1.8 MB)
    |████████████████████████████████| 1.8 MB 2.0 MB/s
...
Installing collected packages: datasette-vega
Attempting uninstall: datasette-vega
    Found existing installation: datasette-vega 0.6
    Uninstalling datasette-vega-0.6:
    Successfully uninstalled datasette-vega-0.6
Successfully installed datasette-vega-0.6.2

$ datasette plugins
[
    {
        "name": "datasette-vega",
        "static": true,
        "templates": false,
        "version": "0.6.2"
    }
]
</code></pre>
<p>I added all of this to the Datasette docs here: <a href="https://docs.datasette.io/en/latest/installation.html#using-pipx" rel="nofollow">https://docs.datasette.io/en/latest/installation.html#using-pipx</a>
(see <a href="https://github.com/simonw/datasette/issues/756">https://github.com/simonw/datasette/issues/756</a>).</p>


path: python_platform-specific-dependencies.md
topic: python
title: Use setup.py to install platform-specific dependencies
url: https://github.com/simonw/til/blob/main/python/platform-specific-dependencies.md
created: 2020-05-05T14:14:36-07:00
created_utc: 2020-05-05T21:14:36+00:00
updated: 2020-05-05T14:14:36-07:00
updated_utc: 2020-05-05T21:14:36+00:00
shot_hash: 60e0d1b3e239b03b42a5a7497d693456
slug: platform-specific-dependencies
body:
For [photos-to-sqlite](https://github.com/dogsheep/photos-to-sqlite) I needed to install `osxphotos` as a dependency, but only if the platform is macOS - it's not available for Linux.

Here's the magic incantation to do that:

```python
setup(
    name="photos-to-sqlite",
    ...
    install_requires=[
        "sqlite-utils>=2.7",
        "boto3>=1.12.41",
        "osxphotos>=0.28.13 ; sys_platform=='darwin'",
    ]
)
```
So ` ; sys_platform=='darwin'` in the install requires line.

More details: https://www.python.org/dev/peps/pep-0508/#environment-markers and https://hynek.me/articles/conditional-python-dependencies/

html:
<p>For <a href="https://github.com/dogsheep/photos-to-sqlite">photos-to-sqlite</a> I needed to install <code>osxphotos</code> as a dependency, but only if the platform is macOS - it's not available for Linux.</p>
<p>Here's the magic incantation to do that:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-en">setup</span>(
    <span class="pl-s1">name</span><span class="pl-c1">=</span><span class="pl-s">"photos-to-sqlite"</span>,
    ...
    <span class="pl-s1">install_requires</span><span class="pl-c1">=</span>[
        <span class="pl-s">"sqlite-utils&gt;=2.7"</span>,
        <span class="pl-s">"boto3&gt;=1.12.41"</span>,
        <span class="pl-s">"osxphotos&gt;=0.28.13 ; sys_platform=='darwin'"</span>,
    ]
)</pre></div>
<p>So <code> ; sys_platform=='darwin'</code> in the install requires line.</p>
<p>More details: <a href="https://www.python.org/dev/peps/pep-0508/#environment-markers" rel="nofollow">https://www.python.org/dev/peps/pep-0508/#environment-markers</a> and <a href="https://hynek.me/articles/conditional-python-dependencies/" rel="nofollow">https://hynek.me/articles/conditional-python-dependencies/</a></p>


path: python_generate-nested-json-summary.md
topic: python
title: Generated a summary of nested JSON data
url: https://github.com/simonw/til/blob/main/python/generate-nested-json-summary.md
created: 2020-04-28T18:07:16-07:00
created_utc: 2020-04-29T01:07:16+00:00
updated: 2020-04-28T18:48:44-07:00
updated_utc: 2020-04-29T01:48:44+00:00
shot_hash: 8afaeb6f21d37a3a389a423865dc2ddb
slug: generate-nested-json-summary
body:
I was trying to figure out the shape of the JSON object from https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data_latest.json?raw=true - which is 3.2MB and heavily nested, so it's difficult to get a good feel for the shape.

I solved this with a Python `summarize()` function which recursively truncates the nested lists and dictionaries.

```python
def summarize(data, list_limit=5, key_limit=5):
    "Recursively reduce data to just the first X nested keys and list items"
    if not isinstance(data, (list, dict)):
        return data
    if isinstance(data, list):
        return [summarize(item, list_limit, key_limit) for item in data[:list_limit]]
    if isinstance(data, dict):
        all_keys = list(data.keys())
        kept_keys = all_keys[:key_limit]
        truncated_keys = all_keys[key_limit:]
        d = dict([
            (key, summarize(data[key], list_limit, key_limit))
            for key in kept_keys
        ])
        if truncated_keys:
            d["_truncated_keys"] = truncated_keys
        return d
```
Here's how I used it:
```python
import json, requests
data = requests.get(
    "https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data_latest.json?raw=true"
).json()
print(json.dumps(summarize(data, list_limit=2, key_limit=7), indent=4))
```
And the output:
```json
{
    "lastUpdatedAt": "2020-04-28T18:14:22.840234Z",
    "disclaimer": "Lab-confirmed case counts for England and subnational areas are provided by Public Health England. All data on deaths and data for the rest of the UK are provided by the Department of Health and Social Care based on data from NHS England and the devolved administrations. Maps include Ordnance Survey data \u00a9 Crown copyright and database right 2020 and Office for National Statistics data \u00a9 Crown copyright and database right 2020. Daily and total case counts are as of 28 April 2020, Daily and total deaths are as of 27 April 2020. See the About the data page (link at top of this page) for details.",
    "overview": {
        "K02000001": {
            "name": {
                "value": "United Kingdom"
            },
            "totalCases": {
                "value": 161145
            },
            "newCases": {
                "value": 3996
            },
            "deaths": {
                "value": 21678
            },
            "dailyDeaths": [
                {
                    "date": "2020-03-15",
                    "value": 14
                },
                {
                    "date": "2020-03-16",
                    "value": 20
                }
            ],
            "dailyTotalDeaths": [
                {
                    "date": "2020-03-10",
                    "value": 6
                },
                {
                    "date": "2020-03-11",
                    "value": 6
                }
            ]
        }
    },
    "countries": {
        "E92000001": {
            "name": {
                "value": "England"
            },
            "totalCases": {
                "value": 114456
            },
            "deaths": {
                "value": 19294
            },
            "maleCases": [
                {
                    "age": "30_to_34",
                    "value": 2224
                },
                {
                    "age": "40_to_44",
                    "value": 2702
                }
            ],
            "femaleCases": [
                {
                    "age": "20_to_24",
                    "value": 1885
                },
                {
                    "age": "90_to_94",
                    "value": 3580
                }
            ],
            "dailyConfirmedCases": [
                {
                    "date": "2020-01-30",
                    "value": 1
                },
                {
                    "date": "2020-01-31",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-01-30",
                    "value": 1
                },
                {
                    "date": "2020-01-31",
                    "value": 2
                }
            ],
            "_truncated_keys": [
                "dailyDeaths",
                "dailyTotalDeaths",
                "previouslyReportedDailyTotalCases",
                "previouslyReportedDailyTotalCasesAdjusted",
                "previouslyReportedDailyCases",
                "previouslyReportedDailyCasesAdjusted",
                "changeInDailyTotalCases",
                "changeInDailyTotalCasesAdjusted",
                "changeInDailyCases",
                "changeInDailyCasesAdjusted"
            ]
        },
        "N92000002": {
            "name": {
                "value": "Northern Ireland"
            },
            "totalCases": {
                "value": 3408
            },
            "deaths": {
                "value": 309
            },
            "dailyDeaths": [
                {
                    "date": "2020-03-28",
                    "value": 2
                },
                {
                    "date": "2020-03-29",
                    "value": 0
                }
            ],
            "dailyTotalDeaths": [
                {
                    "date": "2020-03-27",
                    "value": 13
                },
                {
                    "date": "2020-03-28",
                    "value": 15
                }
            ]
        },
        "S92000003": {
            "name": {
                "value": "Scotland"
            },
            "totalCases": {
                "value": 10721
            },
            "deaths": {
                "value": 1262
            },
            "dailyDeaths": [
                {
                    "date": "2020-03-28",
                    "value": 7
                },
                {
                    "date": "2020-03-29",
                    "value": 0
                }
            ],
            "dailyTotalDeaths": [
                {
                    "date": "2020-03-27",
                    "value": 33
                },
                {
                    "date": "2020-03-28",
                    "value": 40
                }
            ]
        },
        "W92000004": {
            "name": {
                "value": "Wales"
            },
            "totalCases": {
                "value": 9512
            },
            "deaths": {
                "value": 813
            },
            "dailyDeaths": [
                {
                    "date": "2020-03-28",
                    "value": 4
                },
                {
                    "date": "2020-03-29",
                    "value": 10
                }
            ],
            "dailyTotalDeaths": [
                {
                    "date": "2020-03-27",
                    "value": 34
                },
                {
                    "date": "2020-03-28",
                    "value": 38
                }
            ]
        }
    },
    "regions": {
        "E12000004": {
            "name": {
                "value": "East Midlands"
            },
            "totalCases": {
                "value": 6411
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-21",
                    "value": 1
                },
                {
                    "date": "2020-02-25",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-21",
                    "value": 1
                },
                {
                    "date": "2020-02-25",
                    "value": 2
                }
            ]
        },
        "E12000006": {
            "name": {
                "value": "East of England"
            },
            "totalCases": {
                "value": 9907
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 1
                },
                {
                    "date": "2020-02-28",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 1
                },
                {
                    "date": "2020-02-28",
                    "value": 2
                }
            ]
        },
        "E12000007": {
            "name": {
                "value": "London"
            },
            "totalCases": {
                "value": 23979
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-11",
                    "value": 1
                },
                {
                    "date": "2020-02-13",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-11",
                    "value": 1
                },
                {
                    "date": "2020-02-13",
                    "value": 2
                }
            ]
        },
        "E12000001": {
            "name": {
                "value": "North East"
            },
            "totalCases": {
                "value": 7174
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-02",
                    "value": 1
                },
                {
                    "date": "2020-03-04",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-02",
                    "value": 1
                },
                {
                    "date": "2020-03-04",
                    "value": 2
                }
            ]
        },
        "E12000002": {
            "name": {
                "value": "North West"
            },
            "totalCases": {
                "value": 17823
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-28",
                    "value": 1
                },
                {
                    "date": "2020-03-01",
                    "value": 8
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-28",
                    "value": 1
                },
                {
                    "date": "2020-03-01",
                    "value": 9
                }
            ]
        },
        "E12000008": {
            "name": {
                "value": "South East"
            },
            "totalCases": {
                "value": 16323
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-01-31",
                    "value": 1
                },
                {
                    "date": "2020-02-03",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-01-31",
                    "value": 1
                },
                {
                    "date": "2020-02-03",
                    "value": 2
                }
            ]
        },
        "E12000009": {
            "name": {
                "value": "South West"
            },
            "totalCases": {
                "value": 5986
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 2
                },
                {
                    "date": "2020-02-26",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 2
                },
                {
                    "date": "2020-02-26",
                    "value": 3
                }
            ]
        },
        "_truncated_keys": [
            "E12000005",
            "E12000003"
        ]
    },
    "utlas": {
        "E09000002": {
            "name": {
                "value": "Barking and Dagenham"
            },
            "totalCases": {
                "value": 445
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-01",
                    "value": 1
                },
                {
                    "date": "2020-03-08",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-01",
                    "value": 1
                },
                {
                    "date": "2020-03-08",
                    "value": 2
                }
            ]
        },
        "E09000003": {
            "name": {
                "value": "Barnet"
            },
            "totalCases": {
                "value": 1170
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-16",
                    "value": 1
                },
                {
                    "date": "2020-02-28",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-16",
                    "value": 1
                },
                {
                    "date": "2020-02-28",
                    "value": 2
                }
            ]
        },
        "E08000016": {
            "name": {
                "value": "Barnsley"
            },
            "totalCases": {
                "value": 590
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 1
                },
                {
                    "date": "2020-03-02",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-02-03",
                    "value": 1
                },
                {
                    "date": "2020-03-02",
                    "value": 2
                }
            ]
        },
        "E06000022": {
            "name": {
                "value": "Bath and North East Somerset"
            },
            "totalCases": {
                "value": 203
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-11",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 2
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-11",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 3
                }
            ]
        },
        "E06000055": {
            "name": {
                "value": "Bedford"
            },
            "totalCases": {
                "value": 424
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-13",
                    "value": 1
                },
                {
                    "date": "2020-03-17",
                    "value": 2
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-13",
                    "value": 1
                },
                {
                    "date": "2020-03-17",
                    "value": 3
                }
            ]
        },
        "E09000004": {
            "name": {
                "value": "Bexley"
            },
            "totalCases": {
                "value": 596
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-09",
                    "value": 2
                },
                {
                    "date": "2020-03-10",
                    "value": 2
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-09",
                    "value": 2
                },
                {
                    "date": "2020-03-10",
                    "value": 4
                }
            ]
        },
        "E08000025": {
            "name": {
                "value": "Birmingham"
            },
            "totalCases": {
                "value": 2733
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-01",
                    "value": 1
                },
                {
                    "date": "2020-03-02",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-01",
                    "value": 1
                },
                {
                    "date": "2020-03-02",
                    "value": 2
                }
            ]
        },
        "_truncated_keys": [
            "E06000008",
            "E06000009",
            "E08000001",
            "E06000058",
            "E06000036",
            "E08000032",
            "E09000005",
            "E06000043",
            "E06000023",
            "E09000006",
            "E10000002",
            "E08000002",
            "E08000033",
            "E10000003",
            "E09000007",
            "E06000056",
            "E06000049",
            "E06000050",
            "E09000001",
            "E06000052",
            "E06000047",
            "E08000026",
            "E09000008",
            "E10000006",
            "E06000005",
            "E06000015",
            "E10000007",
            "E10000008",
            "E08000017",
            "E06000059",
            "E08000027",
            "E09000009",
            "E06000011",
            "E10000011",
            "E09000010",
            "E10000012",
            "E08000037",
            "E10000013",
            "E09000011",
            "E09000012",
            "E06000006",
            "E09000013",
            "E10000014",
            "E09000014",
            "E09000015",
            "E06000001",
            "E09000016",
            "E06000019",
            "E10000015",
            "E09000017",
            "E09000018",
            "E06000046",
            "E09000019",
            "E09000020",
            "E10000016",
            "E06000010",
            "E09000021",
            "E08000034",
            "E08000011",
            "E09000022",
            "E10000017",
            "E08000035",
            "E06000016",
            "E10000018",
            "E09000023",
            "E10000019",
            "E08000012",
            "E06000032",
            "E08000003",
            "E06000035",
            "E09000024",
            "E06000002",
            "E06000042",
            "E08000021",
            "E09000025",
            "E10000020",
            "E06000012",
            "E06000013",
            "E06000024",
            "E08000022",
            "E10000023",
            "E10000021",
            "E06000057",
            "E06000018",
            "E10000024",
            "E08000004",
            "E10000025",
            "E06000031",
            "E06000026",
            "E06000044",
            "E06000038",
            "E09000026",
            "E06000003",
            "E09000027",
            "E08000005",
            "E08000018",
            "E06000017",
            "E08000006",
            "E08000028",
            "E08000014",
            "E08000019",
            "E06000051",
            "E06000039",
            "E08000029",
            "E10000027",
            "E06000025",
            "E08000023",
            "E06000045",
            "E06000033",
            "E09000028",
            "E08000013",
            "E10000028",
            "E08000007",
            "E06000004",
            "E06000021",
            "E10000029",
            "E08000024",
            "E10000030",
            "E09000029",
            "E06000030",
            "E08000008",
            "E06000020",
            "E06000034",
            "E06000027",
            "E09000030",
            "E08000009",
            "E08000036",
            "E08000030",
            "E09000031",
            "E09000032",
            "E06000007",
            "E10000031",
            "E06000037",
            "E10000032",
            "E09000033",
            "E08000010",
            "E06000054",
            "E06000040",
            "E08000015",
            "E06000041",
            "E08000031",
            "E10000034",
            "E06000014"
        ]
    },
    "ltlas": {
        "E07000223": {
            "name": {
                "value": "Adur"
            },
            "totalCases": {
                "value": 76
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-19",
                    "value": 1
                },
                {
                    "date": "2020-03-22",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-19",
                    "value": 1
                },
                {
                    "date": "2020-03-22",
                    "value": 2
                }
            ]
        },
        "E07000026": {
            "name": {
                "value": "Allerdale"
            },
            "totalCases": {
                "value": 191
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-12",
                    "value": 1
                },
                {
                    "date": "2020-03-13",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-12",
                    "value": 1
                },
                {
                    "date": "2020-03-13",
                    "value": 2
                }
            ]
        },
        "E07000032": {
            "name": {
                "value": "Amber Valley"
            },
            "totalCases": {
                "value": 133
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-12",
                    "value": 1
                },
                {
                    "date": "2020-03-13",
                    "value": 3
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-12",
                    "value": 1
                },
                {
                    "date": "2020-03-13",
                    "value": 4
                }
            ]
        },
        "E07000224": {
            "name": {
                "value": "Arun"
            },
            "totalCases": {
                "value": 121
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-14",
                    "value": 1
                },
                {
                    "date": "2020-03-18",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-14",
                    "value": 1
                },
                {
                    "date": "2020-03-18",
                    "value": 2
                }
            ]
        },
        "E07000170": {
            "name": {
                "value": "Ashfield"
            },
            "totalCases": {
                "value": 182
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-14",
                    "value": 3
                },
                {
                    "date": "2020-03-15",
                    "value": 3
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-14",
                    "value": 3
                },
                {
                    "date": "2020-03-15",
                    "value": 6
                }
            ]
        },
        "E07000105": {
            "name": {
                "value": "Ashford"
            },
            "totalCases": {
                "value": 403
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-03",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-03",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 2
                }
            ]
        },
        "E07000004": {
            "name": {
                "value": "Aylesbury Vale"
            },
            "totalCases": {
                "value": 301
            },
            "dailyConfirmedCases": [
                {
                    "date": "2020-03-04",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 1
                }
            ],
            "dailyTotalConfirmedCases": [
                {
                    "date": "2020-03-04",
                    "value": 1
                },
                {
                    "date": "2020-03-12",
                    "value": 2
                }
            ]
        },
        "_truncated_keys": [
            "E07000200",
            "E09000002",
            "E09000003",
            "E08000016",
            "E07000027",
            "E07000066",
            "E07000084",
            "E07000171",
            "E06000022",
            "E06000055",
            "E09000004",
            "E08000025",
            "E07000129",
            "E06000008",
            "E06000009",
            "E07000033",
            "E08000001",
            "E07000136",
            "E06000058",
            "E06000036",
            "E08000032",
            "E07000067",
            "E07000143",
            "E09000005",
            "E07000068",
            "E06000043",
            "E06000023",
            "E07000144",
            "E09000006",
            "E07000234",
            "E07000095",
            "E07000172",
            "E07000117",
            "E08000002",
            "E08000033",
            "E07000008",
            "E09000007",
            "E07000192",
            "E07000106",
            "E07000028",
            "E07000069",
            "E06000056",
            "E07000130",
            "E07000070",
            "E07000078",
            "E07000177",
            "E06000049",
            "E06000050",
            "E07000034",
            "E07000225",
            "E07000005",
            "E07000118",
            "E09000001",
            "E07000071",
            "E07000029",
            "E07000150",
            "E06000052",
            "E07000079",
            "E06000047",
            "E08000026",
            "E07000163",
            "E07000226",
            "E09000008",
            "E07000096",
            "E06000005",
            "E07000107",
            "E07000151",
            "E06000015",
            "E07000035",
            "E08000017",
            "E06000059",
            "E07000108",
            "E08000027",
            "E09000009",
            "E07000009",
            "E07000040",
            "E07000085",
            "E07000242",
            "E07000137",
            "E07000152",
            "E06000011",
            "E07000193",
            "E07000244",
            "E07000061",
            "E07000086",
            "E07000030",
            "E07000207",
            "E09000010",
            "E07000072",
            "E07000208",
            "E07000036",
            "E07000041",
            "E07000087",
            "E07000010",
            "E07000112",
            "E07000080",
            "E07000119",
            "E08000037",
            "E07000173",
            "E07000081",
            "E07000088",
            "E07000109",
            "E07000145",
            "E09000011",
            "E07000209",
            "E09000012",
            "E06000006",
            "E07000164",
            "E09000013",
            "E07000131",
            "E09000014",
            "E07000073",
            "E07000165",
            "E09000015",
            "E07000089",
            "E06000001",
            "E07000062",
            "E07000090",
            "E09000016",
            "E06000019",
            "E07000098",
            "E07000037",
            "E09000017",
            "E07000132",
            "E07000227",
            "E09000018",
            "E07000011",
            "E07000120",
            "E07000202",
            "E06000046",
            "E09000019",
            "E09000020",
            "E07000153",
            "E07000146",
            "E06000010",
            "E09000021",
            "E08000034",
            "E08000011",
            "E09000022",
            "E07000121",
            "E08000035",
            "E06000016",
            "E07000063",
            "E09000023",
            "E07000194",
            "E07000138",
            "E08000012",
            "E06000032",
            "E07000110",
            "E07000074",
            "E07000235",
            "E08000003",
            "E07000174",
            "E06000035",
            "E07000133",
            "E07000187",
            "E09000024",
            "E07000042",
            "E07000203",
            "E07000228",
            "E06000002",
            "E06000042",
            "E07000210",
            "E07000091",
            "E07000175",
            "E08000021",
            "E07000195",
            "E09000025",
            "E07000043",
            "E07000038",
            "E06000012",
            "E07000099",
            "E07000139",
            "E06000013",
            "E07000147",
            "E06000024",
            "E08000022",
            "E07000218",
            "E07000134",
            "E07000154",
            "E06000057",
            "E07000148",
            "E06000018",
            "E07000219",
            "E07000135",
            "E08000004",
            "E07000178",
            "E07000122",
            "E06000031",
            "E06000026",
            "E06000044",
            "E07000123",
            "E06000038",
            "E09000026",
            "E06000003",
            "E07000236",
            "E07000211",
            "E07000124",
            "E09000027",
            "E07000166",
            "E08000005",
            "E07000075",
            "E07000125",
            "E07000064",
            "E08000018",
            "E07000220",
            "E07000212",
            "E07000176",
            "E07000092",
            "E06000017",
            "E07000167",
            "E08000006",
            "E08000028",
            "E07000168",
            "E07000188",
            "E08000014",
            "E07000169",
            "E07000111",
            "E08000019",
            "E06000051",
            "E06000039",
            "E08000029",
            "E07000246",
            "E07000006",
            "E07000012",
            "E07000039",
            "E06000025",
            "E07000044",
            "E07000140",
            "E07000141",
            "E07000031",
            "E07000149",
            "E07000155",
            "E07000179",
            "E07000126",
            "E07000189",
            "E07000196",
            "E08000023",
            "E06000045",
            "E06000033",
            "E09000028",
            "E07000213",
            "E07000240",
            "E08000013",
            "E07000197",
            "E07000198",
            "E07000243",
            "E08000007",
            "E06000004",
            "E06000021",
            "E07000221",
            "E07000082",
            "E08000024",
            "E07000214",
            "E09000029",
            "E07000113",
            "E06000030",
            "E08000008",
            "E07000199",
            "E07000215",
            "E07000045",
            "E06000020",
            "E07000076",
            "E07000093",
            "E07000083",
            "E07000114",
            "E07000102",
            "E06000034",
            "E07000115",
            "E06000027",
            "E07000046",
            "E09000030",
            "E08000009",
            "E07000116",
            "E07000077",
            "E07000180",
            "E08000036",
            "E08000030",
            "E09000031",
            "E09000032",
            "E06000007",
            "E07000222",
            "E07000103",
            "E07000216",
            "E07000065",
            "E07000156",
            "E07000241",
            "E06000037",
            "E07000047",
            "E07000127",
            "E07000142",
            "E07000181",
            "E07000245",
            "E09000033",
            "E08000010",
            "E06000054",
            "E07000094",
            "E06000040",
            "E08000015",
            "E07000217",
            "E06000041",
            "E08000031",
            "E07000237",
            "E07000229",
            "E07000238",
            "E07000007",
            "E07000128",
            "E07000239",
            "E06000014"
        ]
    }
}
```

html:
<p>I was trying to figure out the shape of the JSON object from <a href="https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data_latest.json?raw=true">https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data_latest.json?raw=true</a> - which is 3.2MB and heavily nested, so it's difficult to get a good feel for the shape.</p>
<p>I solved this with a Python <code>summarize()</code> function which recursively truncates the nested lists and dictionaries.</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">summarize</span>(<span class="pl-s1">data</span>, <span class="pl-s1">list_limit</span><span class="pl-c1">=</span><span class="pl-c1">5</span>, <span class="pl-s1">key_limit</span><span class="pl-c1">=</span><span class="pl-c1">5</span>):
    <span class="pl-s">"Recursively reduce data to just the first X nested keys and list items"</span>
    <span class="pl-k">if</span> <span class="pl-c1">not</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">data</span>, (<span class="pl-s1">list</span>, <span class="pl-s1">dict</span>)):
        <span class="pl-k">return</span> <span class="pl-s1">data</span>
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">data</span>, <span class="pl-s1">list</span>):
        <span class="pl-k">return</span> [<span class="pl-en">summarize</span>(<span class="pl-s1">item</span>, <span class="pl-s1">list_limit</span>, <span class="pl-s1">key_limit</span>) <span class="pl-k">for</span> <span class="pl-s1">item</span> <span class="pl-c1">in</span> <span class="pl-s1">data</span>[:<span class="pl-s1">list_limit</span>]]
    <span class="pl-k">if</span> <span class="pl-en">isinstance</span>(<span class="pl-s1">data</span>, <span class="pl-s1">dict</span>):
        <span class="pl-s1">all_keys</span> <span class="pl-c1">=</span> <span class="pl-en">list</span>(<span class="pl-s1">data</span>.<span class="pl-en">keys</span>())
        <span class="pl-s1">kept_keys</span> <span class="pl-c1">=</span> <span class="pl-s1">all_keys</span>[:<span class="pl-s1">key_limit</span>]
        <span class="pl-s1">truncated_keys</span> <span class="pl-c1">=</span> <span class="pl-s1">all_keys</span>[<span class="pl-s1">key_limit</span>:]
        <span class="pl-s1">d</span> <span class="pl-c1">=</span> <span class="pl-en">dict</span>([
            (<span class="pl-s1">key</span>, <span class="pl-en">summarize</span>(<span class="pl-s1">data</span>[<span class="pl-s1">key</span>], <span class="pl-s1">list_limit</span>, <span class="pl-s1">key_limit</span>))
            <span class="pl-k">for</span> <span class="pl-s1">key</span> <span class="pl-c1">in</span> <span class="pl-s1">kept_keys</span>
        ])
        <span class="pl-k">if</span> <span class="pl-s1">truncated_keys</span>:
            <span class="pl-s1">d</span>[<span class="pl-s">"_truncated_keys"</span>] <span class="pl-c1">=</span> <span class="pl-s1">truncated_keys</span>
        <span class="pl-k">return</span> <span class="pl-s1">d</span></pre></div>
<p>Here's how I used it:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">json</span>, <span class="pl-s1">requests</span>
<span class="pl-s1">data</span> <span class="pl-c1">=</span> <span class="pl-s1">requests</span>.<span class="pl-en">get</span>(
    <span class="pl-s">"https://github.com/simonw/coronavirus-data-gov-archive/blob/master/data_latest.json?raw=true"</span>
).<span class="pl-en">json</span>()
<span class="pl-en">print</span>(<span class="pl-s1">json</span>.<span class="pl-en">dumps</span>(<span class="pl-en">summarize</span>(<span class="pl-s1">data</span>, <span class="pl-s1">list_limit</span><span class="pl-c1">=</span><span class="pl-c1">2</span>, <span class="pl-s1">key_limit</span><span class="pl-c1">=</span><span class="pl-c1">7</span>), <span class="pl-s1">indent</span><span class="pl-c1">=</span><span class="pl-c1">4</span>))</pre></div>
<p>And the output:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>lastUpdatedAt<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-04-28T18:14:22.840234Z<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>disclaimer<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Lab-confirmed case counts for England and subnational areas are provided by Public Health England. All data on deaths and data for the rest of the UK are provided by the Department of Health and Social Care based on data from NHS England and the devolved administrations. Maps include Ordnance Survey data <span class="pl-cce">\u00a9</span> Crown copyright and database right 2020 and Office for National Statistics data <span class="pl-cce">\u00a9</span> Crown copyright and database right 2020. Daily and total case counts are as of 28 April 2020, Daily and total deaths are as of 27 April 2020. See the About the data page (link at top of this page) for details.<span class="pl-pds">"</span></span>,
    <span class="pl-s"><span class="pl-pds">"</span>overview<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>K02000001<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>United Kingdom<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">161145</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>newCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3996</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>deaths<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">21678</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-15<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">14</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-16<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">20</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-10<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">6</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">6</span>
                }
            ]
        }
    },
    <span class="pl-s"><span class="pl-pds">"</span>countries<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>E92000001<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>England<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">114456</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>deaths<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">19294</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>maleCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>30_to_34<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2224</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>40_to_44<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2702</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>femaleCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>20_to_24<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1885</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>age<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>90_to_94<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3580</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-30<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-31<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-30<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-31<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>_truncated_keys<span class="pl-pds">"</span></span>: [
                <span class="pl-s"><span class="pl-pds">"</span>dailyDeaths<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>dailyTotalDeaths<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>previouslyReportedDailyTotalCases<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>previouslyReportedDailyTotalCasesAdjusted<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>previouslyReportedDailyCases<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>previouslyReportedDailyCasesAdjusted<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>changeInDailyTotalCases<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>changeInDailyTotalCasesAdjusted<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>changeInDailyCases<span class="pl-pds">"</span></span>,
                <span class="pl-s"><span class="pl-pds">"</span>changeInDailyCasesAdjusted<span class="pl-pds">"</span></span>
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>N92000002<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Northern Ireland<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3408</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>deaths<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">309</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-29<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">0</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-27<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">13</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">15</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>S92000003<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Scotland<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">10721</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>deaths<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1262</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">7</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-29<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">0</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-27<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">33</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">40</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>W92000004<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Wales<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">9512</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>deaths<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">813</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">4</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-29<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">10</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalDeaths<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-27<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">34</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">38</span>
                }
            ]
        }
    },
    <span class="pl-s"><span class="pl-pds">"</span>regions<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>E12000004<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>East Midlands<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">6411</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-21<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-25<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-21<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-25<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000006<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>East of England<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">9907</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000007<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>London<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">23979</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000001<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North East<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">7174</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000002<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>North West<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">17823</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">8</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">9</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000008<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>South East<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">16323</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-31<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-01-31<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E12000009<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>South West<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">5986</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-26<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-26<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>_truncated_keys<span class="pl-pds">"</span></span>: [
            <span class="pl-s"><span class="pl-pds">"</span>E12000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E12000003<span class="pl-pds">"</span></span>
        ]
    },
    <span class="pl-s"><span class="pl-pds">"</span>utlas<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>E09000002<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Barking and Dagenham<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">445</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-08<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-08<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E09000003<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Barnet<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1170</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-16<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-16<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-28<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E08000016<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Barnsley<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">590</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-02-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E06000022<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Bath and North East Somerset<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">203</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-11<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E06000055<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Bedford<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">424</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-17<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-17<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E09000004<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Bexley<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">596</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-09<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-10<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-09<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-10<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">4</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E08000025<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Birmingham<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2733</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-01<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-02<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>_truncated_keys<span class="pl-pds">"</span></span>: [
            <span class="pl-s"><span class="pl-pds">"</span>E06000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000058<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000036<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000043<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000056<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000049<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000050<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000052<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000047<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000059<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000037<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000014<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000014<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000046<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000020<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000022<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000035<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000035<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000042<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000020<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000022<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000057<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000004<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000044<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000038<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000014<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000051<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000039<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000045<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000004<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000020<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000036<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000037<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000054<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000040<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000041<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E10000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000014<span class="pl-pds">"</span></span>
        ]
    },
    <span class="pl-s"><span class="pl-pds">"</span>ltlas<span class="pl-pds">"</span></span>: {
        <span class="pl-s"><span class="pl-pds">"</span>E07000223<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Adur<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">76</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-19<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-22<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-19<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-22<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000026<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Allerdale<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">191</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000032<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Amber Valley<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">133</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-13<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">4</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000224<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Arun<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">121</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-14<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-18<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-14<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-18<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000170<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Ashfield<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">182</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-14<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-15<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-14<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">3</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-15<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">6</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000105<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Ashford<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">403</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-03<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>E07000004<span class="pl-pds">"</span></span>: {
            <span class="pl-s"><span class="pl-pds">"</span>name<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>Aylesbury Vale<span class="pl-pds">"</span></span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>totalCases<span class="pl-pds">"</span></span>: {
                <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">301</span>
            },
            <span class="pl-s"><span class="pl-pds">"</span>dailyConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                }
            ],
            <span class="pl-s"><span class="pl-pds">"</span>dailyTotalConfirmedCases<span class="pl-pds">"</span></span>: [
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-04<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">1</span>
                },
                {
                    <span class="pl-s"><span class="pl-pds">"</span>date<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>2020-03-12<span class="pl-pds">"</span></span>,
                    <span class="pl-s"><span class="pl-pds">"</span>value<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>
                }
            ]
        },
        <span class="pl-s"><span class="pl-pds">"</span>_truncated_keys<span class="pl-pds">"</span></span>: [
            <span class="pl-s"><span class="pl-pds">"</span>E07000200<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000066<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000084<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000171<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000022<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000055<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000004<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000129<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000136<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000058<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000036<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000067<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000143<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000068<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000043<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000144<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000234<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000095<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000172<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000117<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000192<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000106<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000069<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000056<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000130<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000070<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000078<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000177<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000049<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000050<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000225<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000118<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000071<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000150<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000052<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000079<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000047<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000163<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000226<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000096<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000107<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000151<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000035<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000059<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000108<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000040<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000085<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000242<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000137<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000152<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000193<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000244<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000061<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000086<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000207<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000072<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000208<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000036<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000041<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000087<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000112<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000080<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000119<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000037<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000173<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000081<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000088<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000109<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000145<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000209<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000164<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000131<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000014<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000073<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000165<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000089<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000001<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000062<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000090<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000098<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000037<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000132<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000227<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000120<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000202<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000046<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000020<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000153<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000146<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000011<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000022<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000121<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000035<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000016<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000063<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000194<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000138<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000110<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000074<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000235<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000174<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000035<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000133<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000187<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000042<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000203<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000228<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000002<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000042<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000210<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000091<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000175<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000195<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000043<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000038<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000099<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000139<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000147<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000022<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000218<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000134<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000154<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000057<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000148<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000219<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000135<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000004<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000178<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000122<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000044<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000123<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000038<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000026<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000003<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000236<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000211<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000124<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000166<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000005<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000075<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000125<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000064<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000018<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000220<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000212<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000176<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000092<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000017<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000167<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000168<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000188<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000014<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000169<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000111<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000019<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000051<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000039<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000246<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000006<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000012<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000039<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000025<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000044<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000140<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000141<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000149<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000155<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000179<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000126<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000189<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000196<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000023<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000045<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000028<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000213<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000240<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000013<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000197<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000198<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000243<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000004<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000021<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000221<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000082<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000024<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000214<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000029<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000113<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000008<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000199<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000215<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000045<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000020<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000076<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000093<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000083<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000114<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000102<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000034<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000115<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000027<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000046<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000009<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000116<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000077<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000180<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000036<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000030<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000032<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000222<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000103<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000216<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000065<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000156<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000241<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000037<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000047<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000127<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000142<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000181<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000245<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E09000033<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000010<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000054<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000094<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000040<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000015<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000217<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000041<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E08000031<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000237<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000229<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000238<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000007<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000128<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E07000239<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>E06000014<span class="pl-pds">"</span></span>
        ]
    }
}</pre></div>


path: markdown_github-markdown-api.md
topic: markdown
title: Rendering Markdown with the GitHub Markdown API
url: https://github.com/simonw/til/blob/main/markdown/github-markdown-api.md
created: 2020-08-22T22:30:45-07:00
created_utc: 2020-08-23T05:30:45+00:00
updated: 2020-09-02T16:57:01-07:00
updated_utc: 2020-09-02T23:57:01+00:00
shot_hash: bd614d9cdd761aafc348538180330038
slug: github-markdown-api
body:
I wanted to convert the Markdown used in my TILs to HTML, using the exact same configuration as GitHub does. GitHub has a whole load of custom extensions for things like tables and syntax highlighting (see [issue 17](https://github.com/simonw/til/issues/17)).

It turns out they have [an API for this](https://developer.github.com/v3/markdown/)!

```python
import httpx

body = """
# Example

This is example Markdown
"""

response = httpx.post(
    "https://api.github.com/markdown",
    json={
        # mode=gfm woud expand #13 issue links and suchlike
        "mode": "markdown",
        "text": body,
    },
    headers=headers,
)
if response.status_code == 200:
    markdown_as_html = response.text
```

`response.headers` will give you back information that includes rate-limiting details:
```python
{'server': 'GitHub.com',
 'date': 'Sat, 22 Aug 2020 19:20:29 GMT',
 'content-type': 'text/html;charset=utf-8',
 'content-length': '14',
 'status': '200 OK',
 'x-commonmarker-version': '0.21.0',
 'x-ratelimit-limit': '60',
 'x-ratelimit-remaining': '57',
 'x-ratelimit-reset': '1598125911',
 'access-control-expose-headers': 'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset',
 'access-control-allow-origin': '*',
 'strict-transport-security': 'max-age=31536000; includeSubdomains; preload',
 'x-frame-options': 'deny',
 'x-content-type-options': 'nosniff',
 'x-xss-protection': '1; mode=block',
 'referrer-policy': 'origin-when-cross-origin, strict-origin-when-cross-origin',
 'content-security-policy': "default-src 'none'",
 'vary': 'Accept-Encoding, Accept, X-Requested-With',
 'x-github-request-id': 'C9F0:12D7:32CDD4:8064E2:5F416FFD'}
```
## Calling it from JavaScript with fetch()

I built a [JavaScript tool that uses this API](/tools/render-markdown). Here's the key function:

```javascript
async function render(markdown) {
    return (await fetch('https://api.github.com/markdown', {
        method: 'POST',
        headers: {
            'Content-Type': 'application/json'
        },
        body: JSON.stringify({'mode': 'markdown', 'text': markdown})
    })).text();
}
```
Called like this:
```javascript
let rendered = await render(input.value);
```

## Syntax highlighting CSS

Code examples you send to this API will come back something like this:

```html
<div class="highlight highlight-text-xml-svg"><pre>&lt;<span class="pl-ent">svg</span> <span class="pl-e">viewBox</span>=<span class="pl-s"><span class="pl-pds">"</span>0 0 500 100<span class="pl-pds">"</span></span>&gt;
  &lt;<span class="pl-ent">polyline</span>
     <span class="pl-e">fill</span>=<span class="pl-s"><span class="pl-pds">"</span>none<span class="pl-pds">"</span></span>
     <span class="pl-e">stroke</span>=<span class="pl-s"><span class="pl-pds">"</span>#0074d9<span class="pl-pds">"</span></span>
     <span class="pl-e">stroke-width</span>=<span class="pl-s"><span class="pl-pds">"</span>3<span class="pl-pds">"</span></span>
     <span class="pl-e">points</span>=<span class="pl-s"><span class="pl-pds">"</span></span>
<span class="pl-s">       0,120</span>
<span class="pl-s">       20,60</span>
<span class="pl-s">       40,80</span>
<span class="pl-s">       60,20<span class="pl-pds">"</span></span>/&gt;
```
https://github.com/primer/github-syntax-theme-generator has tools for theming these. I like this stylesheet (MIT licensed): https://github.com/primer/github-syntax-light/blob/master/lib/github-light.css - I figured this out in [issue 20](https://github.com/simonw/til/issues/20).

html:
<p>I wanted to convert the Markdown used in my TILs to HTML, using the exact same configuration as GitHub does. GitHub has a whole load of custom extensions for things like tables and syntax highlighting (see <a href="https://github.com/simonw/til/issues/17">issue 17</a>).</p>
<p>It turns out they have <a href="https://developer.github.com/v3/markdown/">an API for this</a>!</p>
<div class="highlight highlight-source-python"><pre><span class="pl-k">import</span> <span class="pl-s1">httpx</span>

<span class="pl-s1">body</span> <span class="pl-c1">=</span> <span class="pl-s">"""</span>
<span class="pl-s"># Example</span>
<span class="pl-s"></span>
<span class="pl-s">This is example Markdown</span>
<span class="pl-s">"""</span>

<span class="pl-s1">response</span> <span class="pl-c1">=</span> <span class="pl-s1">httpx</span>.<span class="pl-en">post</span>(
    <span class="pl-s">"https://api.github.com/markdown"</span>,
    <span class="pl-s1">json</span><span class="pl-c1">=</span>{
        <span class="pl-c"># mode=gfm woud expand #13 issue links and suchlike</span>
        <span class="pl-s">"mode"</span>: <span class="pl-s">"markdown"</span>,
        <span class="pl-s">"text"</span>: <span class="pl-s1">body</span>,
    },
    <span class="pl-s1">headers</span><span class="pl-c1">=</span><span class="pl-s1">headers</span>,
)
<span class="pl-k">if</span> <span class="pl-s1">response</span>.<span class="pl-s1">status_code</span> <span class="pl-c1">==</span> <span class="pl-c1">200</span>:
    <span class="pl-s1">markdown_as_html</span> <span class="pl-c1">=</span> <span class="pl-s1">response</span>.<span class="pl-s1">text</span></pre></div>
<p><code>response.headers</code> will give you back information that includes rate-limiting details:</p>
<div class="highlight highlight-source-python"><pre>{<span class="pl-s">'server'</span>: <span class="pl-s">'GitHub.com'</span>,
 <span class="pl-s">'date'</span>: <span class="pl-s">'Sat, 22 Aug 2020 19:20:29 GMT'</span>,
 <span class="pl-s">'content-type'</span>: <span class="pl-s">'text/html;charset=utf-8'</span>,
 <span class="pl-s">'content-length'</span>: <span class="pl-s">'14'</span>,
 <span class="pl-s">'status'</span>: <span class="pl-s">'200 OK'</span>,
 <span class="pl-s">'x-commonmarker-version'</span>: <span class="pl-s">'0.21.0'</span>,
 <span class="pl-s">'x-ratelimit-limit'</span>: <span class="pl-s">'60'</span>,
 <span class="pl-s">'x-ratelimit-remaining'</span>: <span class="pl-s">'57'</span>,
 <span class="pl-s">'x-ratelimit-reset'</span>: <span class="pl-s">'1598125911'</span>,
 <span class="pl-s">'access-control-expose-headers'</span>: <span class="pl-s">'ETag, Link, Location, Retry-After, X-GitHub-OTP, X-RateLimit-Limit, X-RateLimit-Remaining, X-RateLimit-Reset, X-OAuth-Scopes, X-Accepted-OAuth-Scopes, X-Poll-Interval, X-GitHub-Media-Type, Deprecation, Sunset'</span>,
 <span class="pl-s">'access-control-allow-origin'</span>: <span class="pl-s">'*'</span>,
 <span class="pl-s">'strict-transport-security'</span>: <span class="pl-s">'max-age=31536000; includeSubdomains; preload'</span>,
 <span class="pl-s">'x-frame-options'</span>: <span class="pl-s">'deny'</span>,
 <span class="pl-s">'x-content-type-options'</span>: <span class="pl-s">'nosniff'</span>,
 <span class="pl-s">'x-xss-protection'</span>: <span class="pl-s">'1; mode=block'</span>,
 <span class="pl-s">'referrer-policy'</span>: <span class="pl-s">'origin-when-cross-origin, strict-origin-when-cross-origin'</span>,
 <span class="pl-s">'content-security-policy'</span>: <span class="pl-s">"default-src 'none'"</span>,
 <span class="pl-s">'vary'</span>: <span class="pl-s">'Accept-Encoding, Accept, X-Requested-With'</span>,
 <span class="pl-s">'x-github-request-id'</span>: <span class="pl-s">'C9F0:12D7:32CDD4:8064E2:5F416FFD'</span>}</pre></div>
<h2>
<a id="user-content-calling-it-from-javascript-with-fetch" class="anchor" href="#calling-it-from-javascript-with-fetch" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Calling it from JavaScript with fetch()</h2>
<p>I built a <a href="/tools/render-markdown">JavaScript tool that uses this API</a>. Here's the key function:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">async</span> <span class="pl-k">function</span> <span class="pl-en">render</span><span class="pl-kos">(</span><span class="pl-s1">markdown</span><span class="pl-kos">)</span> <span class="pl-kos">{</span>
    <span class="pl-k">return</span> <span class="pl-kos">(</span><span class="pl-k">await</span> <span class="pl-en">fetch</span><span class="pl-kos">(</span><span class="pl-s">'https://api.github.com/markdown'</span><span class="pl-kos">,</span> <span class="pl-kos">{</span>
        <span class="pl-c1">method</span>: <span class="pl-s">'POST'</span><span class="pl-kos">,</span>
        <span class="pl-c1">headers</span>: <span class="pl-kos">{</span>
            <span class="pl-s">'Content-Type'</span>: <span class="pl-s">'application/json'</span>
        <span class="pl-kos">}</span><span class="pl-kos">,</span>
        <span class="pl-c1">body</span>: <span class="pl-c1">JSON</span><span class="pl-kos">.</span><span class="pl-en">stringify</span><span class="pl-kos">(</span><span class="pl-kos">{</span><span class="pl-s">'mode'</span>: <span class="pl-s">'markdown'</span><span class="pl-kos">,</span> <span class="pl-s">'text'</span>: <span class="pl-s1">markdown</span><span class="pl-kos">}</span><span class="pl-kos">)</span>
    <span class="pl-kos">}</span><span class="pl-kos">)</span><span class="pl-kos">)</span><span class="pl-kos">.</span><span class="pl-en">text</span><span class="pl-kos">(</span><span class="pl-kos">)</span><span class="pl-kos">;</span>
<span class="pl-kos">}</span></pre></div>
<p>Called like this:</p>
<div class="highlight highlight-source-js"><pre><span class="pl-k">let</span> <span class="pl-s1">rendered</span> <span class="pl-c1">=</span> <span class="pl-k">await</span> <span class="pl-en">render</span><span class="pl-kos">(</span><span class="pl-s1">input</span><span class="pl-kos">.</span><span class="pl-c1">value</span><span class="pl-kos">)</span><span class="pl-kos">;</span></pre></div>
<h2>
<a id="user-content-syntax-highlighting-css" class="anchor" href="#syntax-highlighting-css" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Syntax highlighting CSS</h2>
<p>Code examples you send to this API will come back something like this:</p>
<div class="highlight highlight-text-html-basic"><pre><span class="pl-kos">&lt;</span><span class="pl-ent">div</span> <span class="pl-c1">class</span>="<span class="pl-s">highlight highlight-text-xml-svg</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">pre</span><span class="pl-kos">&gt;</span>&amp;lt;<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-ent</span>"<span class="pl-kos">&gt;</span>svg<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span> <span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-e</span>"<span class="pl-kos">&gt;</span>viewBox<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>=<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>0 0 500 100<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>&amp;gt;
  &amp;lt;<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-ent</span>"<span class="pl-kos">&gt;</span>polyline<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
     <span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-e</span>"<span class="pl-kos">&gt;</span>fill<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>=<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>none<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
     <span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-e</span>"<span class="pl-kos">&gt;</span>stroke<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>=<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>#0074d9<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
     <span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-e</span>"<span class="pl-kos">&gt;</span>stroke-width<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>=<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>3<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
     <span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-e</span>"<span class="pl-kos">&gt;</span>points<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>=<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span><span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span>       0,120<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span>       20,60<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span>       40,80<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>
<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-s</span>"<span class="pl-kos">&gt;</span>       60,20<span class="pl-kos">&lt;</span><span class="pl-ent">span</span> <span class="pl-c1">class</span>="<span class="pl-s">pl-pds</span>"<span class="pl-kos">&gt;</span>"<span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span><span class="pl-kos">&lt;/</span><span class="pl-ent">span</span><span class="pl-kos">&gt;</span>/&amp;gt;</pre></div>
<p><a href="https://github.com/primer/github-syntax-theme-generator">https://github.com/primer/github-syntax-theme-generator</a> has tools for theming these. I like this stylesheet (MIT licensed): <a href="https://github.com/primer/github-syntax-light/blob/master/lib/github-light.css">https://github.com/primer/github-syntax-light/blob/master/lib/github-light.css</a> - I figured this out in <a href="https://github.com/simonw/til/issues/20">issue 20</a>.</p>


path: markdown_converting-to-markdown.md
topic: markdown
title: Converting HTML and rich-text to Markdown
url: https://github.com/simonw/til/blob/main/markdown/converting-to-markdown.md
created: 2020-05-09T11:19:27-07:00
created_utc: 2020-05-09T18:19:27+00:00
updated: 2020-09-01T16:54:03-07:00
updated_utc: 2020-09-01T23:54:03+00:00
shot_hash: 5a052f21f2ce589b08988b6f4bc64674
slug: converting-to-markdown
body:
If you copy and paste from a web page - including a full table - into a GitHub issue comment GitHub converts it to the corresponding Markdown for you. Really quick way to construct Markdown tables.

![GitHub converting to Markdown](https://raw.githubusercontent.com/simonw/til/main/markdown/converting-to-markdown.gif)

https://domchristie.github.io/turndown/ is an open source JavaScript library by Dom Christie that can convert HTML strings into Markdown strings. Code: https://github.com/domchristie/turndown - it used to be called `to-markdown`.

https://euangoddard.github.io/clipboard2markdown/ is a tool which lets you paste in rich-text and uses turndown to convert it for you directly in your browser.

## 2md, explained

https://2md.ca/ is another tool that does this, written in TypeScript. It's accompanied by [a detailed essay](https://2md.ca/how-it-works) explaining exactly how it works.

html:
<p>If you copy and paste from a web page - including a full table - into a GitHub issue comment GitHub converts it to the corresponding Markdown for you. Really quick way to construct Markdown tables.</p>
<p><a href="https://raw.githubusercontent.com/simonw/til/main/markdown/converting-to-markdown.gif" target="_blank" rel="nofollow"><img src="https://raw.githubusercontent.com/simonw/til/main/markdown/converting-to-markdown.gif" alt="GitHub converting to Markdown" style="max-width:100%;"></a></p>
<p><a href="https://domchristie.github.io/turndown/" rel="nofollow">https://domchristie.github.io/turndown/</a> is an open source JavaScript library by Dom Christie that can convert HTML strings into Markdown strings. Code: <a href="https://github.com/domchristie/turndown">https://github.com/domchristie/turndown</a> - it used to be called <code>to-markdown</code>.</p>
<p><a href="https://euangoddard.github.io/clipboard2markdown/" rel="nofollow">https://euangoddard.github.io/clipboard2markdown/</a> is a tool which lets you paste in rich-text and uses turndown to convert it for you directly in your browser.</p>
<h2>
<a id="user-content-2md-explained" class="anchor" href="#2md-explained" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>2md, explained</h2>
<p><a href="https://2md.ca/" rel="nofollow">https://2md.ca/</a> is another tool that does this, written in TypeScript. It's accompanied by <a href="https://2md.ca/how-it-works" rel="nofollow">a detailed essay</a> explaining exactly how it works.</p>


path: homebrew_packaging-python-cli-for-homebrew.md
topic: homebrew
title: Packaging a Python CLI tool for Homebrew
url: https://github.com/simonw/til/blob/main/homebrew/packaging-python-cli-for-homebrew.md
created: 2020-08-11T09:49:17-07:00
created_utc: 2020-08-11T16:49:17+00:00
updated: 2020-12-29T13:55:23-08:00
updated_utc: 2020-12-29T21:55:23+00:00
shot_hash: ea3e3852b1b16e47f6b98efccad32c39
slug: packaging-python-cli-for-homebrew
body:
I finally figured out how to package [Datasette](https://github.com/simonw/datasette) for installation with Homebrew. My package was accepted into Homebrew core, which means you can now install it like this:

    brew install datasette

Prior to being accepted, you needed to install it from my own Homebrew tap like this:

    brew tap simonw/datasette
    brew install datasette
    # wait a bit...
    datasette --version

Or you could skip the `tap` step and run this:

    brew install simonw/datasette/datasette

Here's my code that makes this work: https://github.com/simonw/homebrew-datasette

The [Python for Formula Authors](https://docs.brew.sh/Python-for-Formula-Authors) documentation provides useful background.

## Creating a "tap"

Homebrew taps are just naming conventions. Creating a tap is as simple as creating a GitHub repository with the `homebrew-` prefix. `https://github.com/simonw/homebrew-datasette` is the repo that gets tapped when someone runs `brew tap simonw/datasette`.

The repository needs a [Formula/](https://github.com/simonw/homebrew-datasette/tree/main/Formula) folder. This contains your formulas, which are Ruby `.rb` files.

## Creating the formula

The first working version of the `datasette.rb` formula can be seen here: https://github.com/simonw/homebrew-datasette/blob/e6b71b1aa308d7307f75a6458681fe49f5659098/Formula/datasette.rb

The shape of the formula is this:

```ruby
class Datasette < Formula
  include Language::Python::Virtualenv
  desc "An open source multi-tool for exploring and publishing data"
  homepage "https://datasette.io/"
  url "https://files.pythonhosted.org/packages/96/e2/abc76ee41d9895145e43323c591aa77f2b27959deb640278fc1a43f6b222/datasette-0.46.tar.gz"
  version "0.46"
  sha256 "eb5e5dcb8a0957ed1def841108576afb15a38ce61d222bf54a25d827999ad521"

  depends_on "python@3.8"

  resource "aiofiles" do
    url "https://files.pythonhosted.org/packages/2b/64/437053d6a4ba3b3eea1044131a25b458489320cb9609e19ac17261e4dc9b/aiofiles-0.5.0.tar.gz"
    sha256 "98e6bcfd1b50f97db4980e182ddd509b7cc35909e903a8fe50d8849e02d815af"
  end

  # ... many more resource blocks ...

  def install
    virtualenv_install_with_resources
  end

  test do
    system bin/"datasette", "--help"
  end
end
```

For the initial block you need the URL to a source distribution and the `sha256` code for it. I got these for Datasette by publishing a `sdist` package to PyPI and then clicking [the "View hashes" button](https://pypi.org/project/datasette/0.46/#copy-hash-modal-372b7614-26f5-4356-a89a-3854323983bf) next to the source release.

Every dependency needs to be listed as a resource. They all need to be available as `sdist` packages - I made sure all of my dependencies had an `sdist` on PyPI.

Then I used the [homebrew-pypi-poet](https://github.com/tdsmith/homebrew-pypi-poet) tool to construct the rest of the resource blocks for me. You install that in a fresh virtual environment with the module you are packaging:

```
$ cd /tmp
$ mkdir d
$ cd d
$ pipenv shell
(d) $ pip install datasette homebrew-pypi-poet
...
(d) $ poet 
usage: poet [-h] [--single package [package ...] | --formula package | --resources package] [--also package] [-V]
(d) $ poet datasette
  resource "aiofiles" do
    url "https://files.pythonhosted.org/packages/2b/64/437053d6a4ba3b3eea1044131a25b458489320cb9609e19ac17261e4dc9b/aiofiles-0.5.0.tar.gz"
    sha256 "98e6bcfd1b50f97db4980e182ddd509b7cc35909e903a8fe50d8849e02d815af"
  end

  resource "asgi-csrf" do
    url "https://files.pythonhosted.org/packages/20/19/60188a9d88e5af17cfb91dac465f898d8eccf69bc215ac731d64c49fea5c/asgi-csrf-0.6.1.tar.gz"
    sha256 "4045b8b45c330e068b8b96f914e585ea69228efbfe574ab4a4be2d8c6009a19f"
  end
  ...
```
There's one major gotcha here: `datasette` itself was included as a `resource` block. It's important to *remove this resource block* - if you don't, the `datasette` command-line tool will not be sym-linked from the `/usr/local/bin` directory by Homebrew which means users won't be able to type `datasette` to run it. See [issue #2](https://github.com/simonw/homebrew-datasette/issues/2) for how we figured that out.

But that's it! Publish the new `yourname.rb` file, run the `brew tap yourname/yourtap` and `brew install yourname/yourtap/yourformula`.

## Even better: poet -f

Running `poet datasette` generates the resource stanzas, but leaves you to add the rest of the formula (and manually remove the package itself from that list of resources).

`poet -f datasette` generates the full formula.

You need to fill in the description and the `test` block, but other than that it looks like it should work straight away.

## Implementing the test block

https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula says:

> We want tests that don't require any user input and test the basic functionality of the application. For example `foo build-foo input.foo` is a good test and (despite their widespread use) `foo --version` and `foo --help` are bad tests. However, a bad test is better than no test at all.

Here's the test block I ended up using for Datasette:

```ruby
  test do
    assert_match "15", shell_output("#{bin}/datasette --get '/:memory:.csv?sql=select+3*5'")
    assert_match "<title>Datasette:", shell_output("#{bin}/datasette --get '/'")
  end
```

And here's my test for `sqlite-utils`:

```ruby
  test do
    assert_match "15", shell_output("#{bin}/sqlite-utils :memory: 'select 3 * 5'")
  end
```

## Iterating on this

I found running `brew install datasette`, seeing if it worked, then running `brew uninstall datasette`, modifying the `.rb` file on GitHub and running `datasette install datasette` again worked fine during development.

If you get any errors, `brew install datasette --debug` shows more information and drops you into an interactive debugging session when an error occurs.

## Submitting to homebrew-core

If your package gets accepted into [homebrew-core](https://github.com/Homebrew/homebrew-core) users will be able to install it just by running `brew install packagename`.

More importantly: Homebrew maintain "bottle" versions of all of those core packages. These are pre-compiled bundles of assets (a separate `.tar.gz` for each recent macOS operating system) which install MUCH faster than regular Homebrew, which has to compile everything.

The Homebrew [CONTRIBUTING](https://github.com/Homebrew/homebrew-core/blob/master/CONTRIBUTING.md) document tells you how to do this. For Python packages the import things to remember are:

- Add a `license`, e.g. `license "Apache 2.0" - [example](https://github.com/Homebrew/homebrew-core/blob/99c3304fbe89996ae8d72b5357b14fbe8983680c/Formula/datasette.rb#L7).
- Run `brew audit --new-formula datasette` and fix any warnings ([see here](https://github.com/simonw/homebrew-datasette/issues/7)).
- Submit a PR with the new formula and a title of e.g. `datasette 0.47.1 (new formula)` - here's [mine for Datasette](https://github.com/Homebrew/homebrew-core/pull/59494) and [for sqlite-utils](https://github.com/Homebrew/homebrew-core/pull/59533).

html:
<p>I finally figured out how to package <a href="https://github.com/simonw/datasette">Datasette</a> for installation with Homebrew. My package was accepted into Homebrew core, which means you can now install it like this:</p>
<pre><code>brew install datasette
</code></pre>
<p>Prior to being accepted, you needed to install it from my own Homebrew tap like this:</p>
<pre><code>brew tap simonw/datasette
brew install datasette
# wait a bit...
datasette --version
</code></pre>
<p>Or you could skip the <code>tap</code> step and run this:</p>
<pre><code>brew install simonw/datasette/datasette
</code></pre>
<p>Here's my code that makes this work: <a href="https://github.com/simonw/homebrew-datasette">https://github.com/simonw/homebrew-datasette</a></p>
<p>The <a href="https://docs.brew.sh/Python-for-Formula-Authors" rel="nofollow">Python for Formula Authors</a> documentation provides useful background.</p>
<h2>
<a id="user-content-creating-a-tap" class="anchor" href="#creating-a-tap" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating a "tap"</h2>
<p>Homebrew taps are just naming conventions. Creating a tap is as simple as creating a GitHub repository with the <code>homebrew-</code> prefix. <code>https://github.com/simonw/homebrew-datasette</code> is the repo that gets tapped when someone runs <code>brew tap simonw/datasette</code>.</p>
<p>The repository needs a <a href="https://github.com/simonw/homebrew-datasette/tree/main/Formula">Formula/</a> folder. This contains your formulas, which are Ruby <code>.rb</code> files.</p>
<h2>
<a id="user-content-creating-the-formula" class="anchor" href="#creating-the-formula" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the formula</h2>
<p>The first working version of the <code>datasette.rb</code> formula can be seen here: <a href="https://github.com/simonw/homebrew-datasette/blob/e6b71b1aa308d7307f75a6458681fe49f5659098/Formula/datasette.rb">https://github.com/simonw/homebrew-datasette/blob/e6b71b1aa308d7307f75a6458681fe49f5659098/Formula/datasette.rb</a></p>
<p>The shape of the formula is this:</p>
<div class="highlight highlight-source-ruby"><pre><span class="pl-k">class</span> <span class="pl-v">Datasette</span> &lt; <span class="pl-v">Formula</span>
  <span class="pl-en">include</span> <span class="pl-v">Language</span>::<span class="pl-v">Python</span>::<span class="pl-v">Virtualenv</span>
  <span class="pl-en">desc</span> <span class="pl-s">"An open source multi-tool for exploring and publishing data"</span>
  <span class="pl-en">homepage</span> <span class="pl-s">"https://datasette.io/"</span>
  <span class="pl-en">url</span> <span class="pl-s">"https://files.pythonhosted.org/packages/96/e2/abc76ee41d9895145e43323c591aa77f2b27959deb640278fc1a43f6b222/datasette-0.46.tar.gz"</span>
  <span class="pl-en">version</span> <span class="pl-s">"0.46"</span>
  <span class="pl-en">sha256</span> <span class="pl-s">"eb5e5dcb8a0957ed1def841108576afb15a38ce61d222bf54a25d827999ad521"</span>

  <span class="pl-en">depends_on</span> <span class="pl-s">"python@3.8"</span>

  <span class="pl-en">resource</span> <span class="pl-s">"aiofiles"</span> <span class="pl-k">do</span>
    <span class="pl-en">url</span> <span class="pl-s">"https://files.pythonhosted.org/packages/2b/64/437053d6a4ba3b3eea1044131a25b458489320cb9609e19ac17261e4dc9b/aiofiles-0.5.0.tar.gz"</span>
    <span class="pl-en">sha256</span> <span class="pl-s">"98e6bcfd1b50f97db4980e182ddd509b7cc35909e903a8fe50d8849e02d815af"</span>
  <span class="pl-k">end</span>

  <span class="pl-c"># ... many more resource blocks ...</span>

  <span class="pl-k">def</span> <span class="pl-en">install</span>
    <span class="pl-en">virtualenv_install_with_resources</span>
  <span class="pl-k">end</span>

  <span class="pl-en">test</span> <span class="pl-k">do</span>
    <span class="pl-en">system</span> <span class="pl-en">bin</span>/<span class="pl-s">"datasette"</span><span class="pl-kos">,</span> <span class="pl-s">"--help"</span>
  <span class="pl-k">end</span>
<span class="pl-k">end</span></pre></div>
<p>For the initial block you need the URL to a source distribution and the <code>sha256</code> code for it. I got these for Datasette by publishing a <code>sdist</code> package to PyPI and then clicking <a href="https://pypi.org/project/datasette/0.46/#copy-hash-modal-372b7614-26f5-4356-a89a-3854323983bf" rel="nofollow">the "View hashes" button</a> next to the source release.</p>
<p>Every dependency needs to be listed as a resource. They all need to be available as <code>sdist</code> packages - I made sure all of my dependencies had an <code>sdist</code> on PyPI.</p>
<p>Then I used the <a href="https://github.com/tdsmith/homebrew-pypi-poet">homebrew-pypi-poet</a> tool to construct the rest of the resource blocks for me. You install that in a fresh virtual environment with the module you are packaging:</p>
<pre><code>$ cd /tmp
$ mkdir d
$ cd d
$ pipenv shell
(d) $ pip install datasette homebrew-pypi-poet
...
(d) $ poet 
usage: poet [-h] [--single package [package ...] | --formula package | --resources package] [--also package] [-V]
(d) $ poet datasette
  resource "aiofiles" do
    url "https://files.pythonhosted.org/packages/2b/64/437053d6a4ba3b3eea1044131a25b458489320cb9609e19ac17261e4dc9b/aiofiles-0.5.0.tar.gz"
    sha256 "98e6bcfd1b50f97db4980e182ddd509b7cc35909e903a8fe50d8849e02d815af"
  end

  resource "asgi-csrf" do
    url "https://files.pythonhosted.org/packages/20/19/60188a9d88e5af17cfb91dac465f898d8eccf69bc215ac731d64c49fea5c/asgi-csrf-0.6.1.tar.gz"
    sha256 "4045b8b45c330e068b8b96f914e585ea69228efbfe574ab4a4be2d8c6009a19f"
  end
  ...
</code></pre>
<p>There's one major gotcha here: <code>datasette</code> itself was included as a <code>resource</code> block. It's important to <em>remove this resource block</em> - if you don't, the <code>datasette</code> command-line tool will not be sym-linked from the <code>/usr/local/bin</code> directory by Homebrew which means users won't be able to type <code>datasette</code> to run it. See <a href="https://github.com/simonw/homebrew-datasette/issues/2">issue #2</a> for how we figured that out.</p>
<p>But that's it! Publish the new <code>yourname.rb</code> file, run the <code>brew tap yourname/yourtap</code> and <code>brew install yourname/yourtap/yourformula</code>.</p>
<h2>
<a id="user-content-even-better-poet--f" class="anchor" href="#even-better-poet--f" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Even better: poet -f</h2>
<p>Running <code>poet datasette</code> generates the resource stanzas, but leaves you to add the rest of the formula (and manually remove the package itself from that list of resources).</p>
<p><code>poet -f datasette</code> generates the full formula.</p>
<p>You need to fill in the description and the <code>test</code> block, but other than that it looks like it should work straight away.</p>
<h2>
<a id="user-content-implementing-the-test-block" class="anchor" href="#implementing-the-test-block" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Implementing the test block</h2>
<p><a href="https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula" rel="nofollow">https://docs.brew.sh/Formula-Cookbook#add-a-test-to-the-formula</a> says:</p>
<blockquote>
<p>We want tests that don't require any user input and test the basic functionality of the application. For example <code>foo build-foo input.foo</code> is a good test and (despite their widespread use) <code>foo --version</code> and <code>foo --help</code> are bad tests. However, a bad test is better than no test at all.</p>
</blockquote>
<p>Here's the test block I ended up using for Datasette:</p>
<div class="highlight highlight-source-ruby"><pre>  <span class="pl-en">test</span> <span class="pl-k">do</span>
    <span class="pl-en">assert_match</span> <span class="pl-s">"15"</span><span class="pl-kos">,</span> <span class="pl-en">shell_output</span><span class="pl-kos">(</span><span class="pl-s">"<span class="pl-s1"><span class="pl-kos">#{</span><span class="pl-en">bin</span><span class="pl-kos">}</span></span>/datasette --get '/:memory:.csv?sql=select+3*5'"</span><span class="pl-kos">)</span>
    <span class="pl-en">assert_match</span> <span class="pl-s">"&lt;title&gt;Datasette:"</span><span class="pl-kos">,</span> <span class="pl-en">shell_output</span><span class="pl-kos">(</span><span class="pl-s">"<span class="pl-s1"><span class="pl-kos">#{</span><span class="pl-en">bin</span><span class="pl-kos">}</span></span>/datasette --get '/'"</span><span class="pl-kos">)</span>
  <span class="pl-k">end</span></pre></div>
<p>And here's my test for <code>sqlite-utils</code>:</p>
<div class="highlight highlight-source-ruby"><pre>  <span class="pl-en">test</span> <span class="pl-k">do</span>
    <span class="pl-en">assert_match</span> <span class="pl-s">"15"</span><span class="pl-kos">,</span> <span class="pl-en">shell_output</span><span class="pl-kos">(</span><span class="pl-s">"<span class="pl-s1"><span class="pl-kos">#{</span><span class="pl-en">bin</span><span class="pl-kos">}</span></span>/sqlite-utils :memory: 'select 3 * 5'"</span><span class="pl-kos">)</span>
  <span class="pl-k">end</span></pre></div>
<h2>
<a id="user-content-iterating-on-this" class="anchor" href="#iterating-on-this" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Iterating on this</h2>
<p>I found running <code>brew install datasette</code>, seeing if it worked, then running <code>brew uninstall datasette</code>, modifying the <code>.rb</code> file on GitHub and running <code>datasette install datasette</code> again worked fine during development.</p>
<p>If you get any errors, <code>brew install datasette --debug</code> shows more information and drops you into an interactive debugging session when an error occurs.</p>
<h2>
<a id="user-content-submitting-to-homebrew-core" class="anchor" href="#submitting-to-homebrew-core" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Submitting to homebrew-core</h2>
<p>If your package gets accepted into <a href="https://github.com/Homebrew/homebrew-core">homebrew-core</a> users will be able to install it just by running <code>brew install packagename</code>.</p>
<p>More importantly: Homebrew maintain "bottle" versions of all of those core packages. These are pre-compiled bundles of assets (a separate <code>.tar.gz</code> for each recent macOS operating system) which install MUCH faster than regular Homebrew, which has to compile everything.</p>
<p>The Homebrew <a href="https://github.com/Homebrew/homebrew-core/blob/master/CONTRIBUTING.md">CONTRIBUTING</a> document tells you how to do this. For Python packages the import things to remember are:</p>
<ul>
<li>Add a <code>license</code>, e.g. `license "Apache 2.0" - <a href="https://github.com/Homebrew/homebrew-core/blob/99c3304fbe89996ae8d72b5357b14fbe8983680c/Formula/datasette.rb#L7">example</a>.</li>
<li>Run <code>brew audit --new-formula datasette</code> and fix any warnings (<a href="https://github.com/simonw/homebrew-datasette/issues/7">see here</a>).</li>
<li>Submit a PR with the new formula and a title of e.g. <code>datasette 0.47.1 (new formula)</code> - here's <a href="https://github.com/Homebrew/homebrew-core/pull/59494">mine for Datasette</a> and <a href="https://github.com/Homebrew/homebrew-core/pull/59533">for sqlite-utils</a>.</li>
</ul>


path: homebrew_homebrew-core-local-git-checkout.md
topic: homebrew
title: Browsing your local git checkout of homebrew-core
url: https://github.com/simonw/til/blob/main/homebrew/homebrew-core-local-git-checkout.md
created: 2020-08-27T17:39:16-07:00
created_utc: 2020-08-28T00:39:16+00:00
updated: 2020-08-27T17:39:16-07:00
updated_utc: 2020-08-28T00:39:16+00:00
shot_hash: e24e0a43dde8ca37b1d2c271279e403e
slug: homebrew-core-local-git-checkout
body:
The [homebrew-core](https://github.com/Homebrew/homebrew-core) repository contains all of the default formulas for Homebrew.

It's a huge repo, and if you browse it through the GitHub web interface you can run into errors like this one:

https://github.com/Homebrew/homebrew-core/commits/master/Formula/libspatialite.rb

> ### Sorry, this commit history is taking too long to generate.
>
> Refresh the page to try again, or view this history locally using the following command:
>
>     git log master -- Formula/libspatialite.rb

It turns out there's a full checkout of the repo (including history) in this folder on your computer already:

    /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core

So you can browse the history for that file locally like so:

    cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core
    git log master -- Formula/libspatialite.rb

html:
<p>The <a href="https://github.com/Homebrew/homebrew-core">homebrew-core</a> repository contains all of the default formulas for Homebrew.</p>
<p>It's a huge repo, and if you browse it through the GitHub web interface you can run into errors like this one:</p>
<p><a href="https://github.com/Homebrew/homebrew-core/commits/master/Formula/libspatialite.rb">https://github.com/Homebrew/homebrew-core/commits/master/Formula/libspatialite.rb</a></p>
<blockquote>
<h3>
<a id="user-content-sorry-this-commit-history-is-taking-too-long-to-generate" class="anchor" href="#sorry-this-commit-history-is-taking-too-long-to-generate" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Sorry, this commit history is taking too long to generate.</h3>
<p>Refresh the page to try again, or view this history locally using the following command:</p>
<pre><code>git log master -- Formula/libspatialite.rb
</code></pre>
</blockquote>
<p>It turns out there's a full checkout of the repo (including history) in this folder on your computer already:</p>
<pre><code>/usr/local/Homebrew/Library/Taps/homebrew/homebrew-core
</code></pre>
<p>So you can browse the history for that file locally like so:</p>
<pre><code>cd /usr/local/Homebrew/Library/Taps/homebrew/homebrew-core
git log master -- Formula/libspatialite.rb
</code></pre>


path: zeit-now_python-asgi-on-now-v2.md
topic: zeit-now
title: Running a Python ASGI app on Vercel
url: https://github.com/simonw/til/blob/main/zeit-now/python-asgi-on-now-v2.md
created: 2020-04-19T15:28:51-07:00
created_utc: 2020-04-19T22:28:51+00:00
updated: 2021-01-18T17:25:03-08:00
updated_utc: 2021-01-19T01:25:03+00:00
shot_hash: 6b5b9adc03ac26bdbd0271754b381696
slug: python-asgi-on-now-v2
body:
Vercel really wants you to deploy static assets with [serverless functions](https://vercel.com/docs/serverless-functions/introduction) tucked away in a separate folder. They suggest creating modules like `api/index.py` which will be served up automatically as API backends to your client-side JavaScript.

It turns out you can subvert that model entirely and route all of your traffic through a single function - great for serving up Python WSGI or ASGI apps that handle traffic routing themselves.

the trick is to use the `"routes"` key in a `now.json` file like this:

```json
{
    "version": 2,
    "builds": [
        {
            "src": "json_head.py",
            "use": "@now/python"
        }
    ],
    "routes": [
        {
            "src": "(.*)",
            "dest": "json_head.py"
        }
    ]
}
```

Here `json_head.py` is a Python module which uses [Sanic](https://github.com/huge-success/sanic) to expose a `app` object that conforms to the ASGI protocol. The same trick works for [Starlette](https://github.com/encode/starlette) too, and should work for libraries that expose WSGI protocol objects such as Flask.

Some examples I've built that use this pattern:

* https://github.com/simonw/json-head
* https://github.com/simonw/gzthermal-web
* Apps published using https://github.com/simonw/datasette-publish-now

Here's [the Vercel builder code](https://github.com/vercel/vercel/blob/c9437e714a754da2d25ae23160d5ad9cf64e2228/packages/now-python/now_init.py#L82) that detects ASGI or WSGI apps, for a peek at how it works under the hood.

See also:

* [Hello, Zeit Now v2](https://simonwillison.net/2020/Apr/8/weeknotes-zeit-now-v2/#hello-zeit-now-v2) where I started exploring it more
* [Better Datasette on Now v2](https://simonwillison.net/2020/Apr/16/weeknotes-hacking-23-different-projects/) documenting further improvement tricks I uncovered

html:
<p>Vercel really wants you to deploy static assets with <a href="https://vercel.com/docs/serverless-functions/introduction" rel="nofollow">serverless functions</a> tucked away in a separate folder. They suggest creating modules like <code>api/index.py</code> which will be served up automatically as API backends to your client-side JavaScript.</p>
<p>It turns out you can subvert that model entirely and route all of your traffic through a single function - great for serving up Python WSGI or ASGI apps that handle traffic routing themselves.</p>
<p>the trick is to use the <code>"routes"</code> key in a <code>now.json</code> file like this:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-s"><span class="pl-pds">"</span>version<span class="pl-pds">"</span></span>: <span class="pl-c1">2</span>,
    <span class="pl-s"><span class="pl-pds">"</span>builds<span class="pl-pds">"</span></span>: [
        {
            <span class="pl-s"><span class="pl-pds">"</span>src<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>json_head.py<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>use<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>@now/python<span class="pl-pds">"</span></span>
        }
    ],
    <span class="pl-s"><span class="pl-pds">"</span>routes<span class="pl-pds">"</span></span>: [
        {
            <span class="pl-s"><span class="pl-pds">"</span>src<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>(.*)<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>dest<span class="pl-pds">"</span></span>: <span class="pl-s"><span class="pl-pds">"</span>json_head.py<span class="pl-pds">"</span></span>
        }
    ]
}</pre></div>
<p>Here <code>json_head.py</code> is a Python module which uses <a href="https://github.com/huge-success/sanic">Sanic</a> to expose a <code>app</code> object that conforms to the ASGI protocol. The same trick works for <a href="https://github.com/encode/starlette">Starlette</a> too, and should work for libraries that expose WSGI protocol objects such as Flask.</p>
<p>Some examples I've built that use this pattern:</p>
<ul>
<li><a href="https://github.com/simonw/json-head">https://github.com/simonw/json-head</a></li>
<li><a href="https://github.com/simonw/gzthermal-web">https://github.com/simonw/gzthermal-web</a></li>
<li>Apps published using <a href="https://github.com/simonw/datasette-publish-now">https://github.com/simonw/datasette-publish-now</a>
</li>
</ul>
<p>Here's <a href="https://github.com/vercel/vercel/blob/c9437e714a754da2d25ae23160d5ad9cf64e2228/packages/now-python/now_init.py#L82">the Vercel builder code</a> that detects ASGI or WSGI apps, for a peek at how it works under the hood.</p>
<p>See also:</p>
<ul>
<li>
<a href="https://simonwillison.net/2020/Apr/8/weeknotes-zeit-now-v2/#hello-zeit-now-v2" rel="nofollow">Hello, Zeit Now v2</a> where I started exploring it more</li>
<li>
<a href="https://simonwillison.net/2020/Apr/16/weeknotes-hacking-23-different-projects/" rel="nofollow">Better Datasette on Now v2</a> documenting further improvement tricks I uncovered</li>
</ul>


path: sqlite_lag-window-function.md
topic: sqlite
title: Lag window function in SQLite
url: https://github.com/simonw/til/blob/main/sqlite/lag-window-function.md
created: 2020-04-19T16:23:04-07:00
created_utc: 2020-04-19T23:23:04+00:00
updated: 2020-04-19T16:23:04-07:00
updated_utc: 2020-04-19T23:23:04+00:00
shot_hash: 358f3838e99fe4c57990325e76d98d7f
slug: lag-window-function
body:
Here's [how to use a lag window function](https://covid-19.datasettes.com/covid?sql=with+italy+as+%28%0D%0A++select%0D%0A++++rowid%2C%0D%0A++++day%2C%0D%0A++++country_or_region%2C%0D%0A++++province_or_state%2C%0D%0A++++admin2%2C%0D%0A++++fips%2C%0D%0A++++confirmed%2C%0D%0A++++deaths%2C%0D%0A++++recovered%2C%0D%0A++++active%2C%0D%0A++++latitude%2C%0D%0A++++longitude%2C%0D%0A++++last_update%2C%0D%0A++++combined_key%0D%0A++from%0D%0A++++johns_hopkins_csse_daily_reports%0D%0A++where%0D%0A++++%22country_or_region%22+%3D+%3Ap0%0D%0A++order+by%0D%0A++++confirmed+desc%0D%0A%29%0D%0Aselect%0D%0A++day%2C%0D%0A++confirmed+-+lag%28confirmed%2C+1%29+OVER+%28%0D%0A++++ORDER+BY%0D%0A++++++day%0D%0A++%29+as+new_cases%0D%0Afrom%0D%0A++italy%0D%0Aorder+by+day+desc+limit+50&p0=Italy#g.mark=bar&g.x_column=day&g.x_type=ordinal&g.y_column=new_cases&g.y_type=quantitative) to calculate new cases per day when the table just has total cases over time on different dates.

The key clause is this:
```sql
select
  day,
  confirmed - lag(confirmed, 1) OVER (
    ORDER BY
      day
  ) as new_cases
```

So the syntax is `lag(column, 1) over (order by day)` - to get the previous value of `column` based on the `day`.

Full example query (using a CTE as well):

```sql
with italy as (
  select
    rowid,
    day,
    country_or_region,
    province_or_state,
    admin2,
    fips,
    confirmed,
    deaths,
    recovered,
    active,
    latitude,
    longitude,
    last_update,
    combined_key
  from
    johns_hopkins_csse_daily_reports
  where
    "country_or_region" = :p0
  order by
    confirmed desc
)
select
  day,
  confirmed - lag(confirmed, 1) OVER (
    ORDER BY
      day
  ) as new_cases
from
  italy
order by day desc limit 50
```

Originally tweeted here: https://twitter.com/simonw/status/1246482954630492200

html:
<p>Here's <a href="https://covid-19.datasettes.com/covid?sql=with+italy+as+%28%0D%0A++select%0D%0A++++rowid%2C%0D%0A++++day%2C%0D%0A++++country_or_region%2C%0D%0A++++province_or_state%2C%0D%0A++++admin2%2C%0D%0A++++fips%2C%0D%0A++++confirmed%2C%0D%0A++++deaths%2C%0D%0A++++recovered%2C%0D%0A++++active%2C%0D%0A++++latitude%2C%0D%0A++++longitude%2C%0D%0A++++last_update%2C%0D%0A++++combined_key%0D%0A++from%0D%0A++++johns_hopkins_csse_daily_reports%0D%0A++where%0D%0A++++%22country_or_region%22+%3D+%3Ap0%0D%0A++order+by%0D%0A++++confirmed+desc%0D%0A%29%0D%0Aselect%0D%0A++day%2C%0D%0A++confirmed+-+lag%28confirmed%2C+1%29+OVER+%28%0D%0A++++ORDER+BY%0D%0A++++++day%0D%0A++%29+as+new_cases%0D%0Afrom%0D%0A++italy%0D%0Aorder+by+day+desc+limit+50&amp;p0=Italy#g.mark=bar&amp;g.x_column=day&amp;g.x_type=ordinal&amp;g.y_column=new_cases&amp;g.y_type=quantitative" rel="nofollow">how to use a lag window function</a> to calculate new cases per day when the table just has total cases over time on different dates.</p>
<p>The key clause is this:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span>
  day,
  confirmed <span class="pl-k">-</span> lag(confirmed, <span class="pl-c1">1</span>) OVER (
    <span class="pl-k">ORDER BY</span>
      day
  ) <span class="pl-k">as</span> new_cases</pre></div>
<p>So the syntax is <code>lag(column, 1) over (order by day)</code> - to get the previous value of <code>column</code> based on the <code>day</code>.</p>
<p>Full example query (using a CTE as well):</p>
<div class="highlight highlight-source-sql"><pre>with italy <span class="pl-k">as</span> (
  <span class="pl-k">select</span>
    rowid,
    day,
    country_or_region,
    province_or_state,
    admin2,
    fips,
    confirmed,
    deaths,
    recovered,
    active,
    latitude,
    longitude,
    last_update,
    combined_key
  <span class="pl-k">from</span>
    johns_hopkins_csse_daily_reports
  <span class="pl-k">where</span>
    <span class="pl-s"><span class="pl-pds">"</span>country_or_region<span class="pl-pds">"</span></span> <span class="pl-k">=</span> :p0
  <span class="pl-k">order by</span>
    confirmed <span class="pl-k">desc</span>
)
<span class="pl-k">select</span>
  day,
  confirmed <span class="pl-k">-</span> lag(confirmed, <span class="pl-c1">1</span>) OVER (
    <span class="pl-k">ORDER BY</span>
      day
  ) <span class="pl-k">as</span> new_cases
<span class="pl-k">from</span>
  italy
<span class="pl-k">order by</span> day <span class="pl-k">desc</span> <span class="pl-k">limit</span> <span class="pl-c1">50</span></pre></div>
<p>Originally tweeted here: <a href="https://twitter.com/simonw/status/1246482954630492200" rel="nofollow">https://twitter.com/simonw/status/1246482954630492200</a></p>


path: sqlite_blob-literals.md
topic: sqlite
title: SQLite BLOB literals
url: https://github.com/simonw/til/blob/main/sqlite/blob-literals.md
created: 2020-07-29T14:49:43-07:00
created_utc: 2020-07-29T21:49:43+00:00
updated: 2020-07-29T15:07:04-07:00
updated_utc: 2020-07-29T22:07:04+00:00
shot_hash: 87668bd639ce7796c4803f0ff8e41110
slug: blob-literals
body:
I wanted to construct a string of SQL that would return a blob value:

```sql
select 'binary-data' as content, 'x.jpg' as content_filename
```

This was while writing a unit test for `datasette-media` - for [issue #19](https://github.com/simonw/datasette-media/issues/19). I used it in the test [here](https://github.com/simonw/datasette-media/blob/2cf64d949ccb8cd5f34b24aeb41b2a91de14cdd2/tests/test_media.py#L292-L295).

The SQLite documentation for [Literal values](https://www.sqlite.org/lang_expr.html#literal_values_constants_) explains how to do this:

> BLOB literals are string literals containing hexadecimal data and preceded by a single "x" or "X" character. Example: X'53514C697465' 

In Python 3 you can generate the hexadecimal representation of any byte string using `b'...'.hex()`

So my solution looked like this:

```python
jpeg_bytes = open("content.jpg", "rb").read()
sql = "select X'{}' as content, 'x.jpg' as content_filename".format(jpeg_bytes.hex())
```

html:
<p>I wanted to construct a string of SQL that would return a blob value:</p>
<div class="highlight highlight-source-sql"><pre><span class="pl-k">select</span> <span class="pl-s"><span class="pl-pds">'</span>binary-data<span class="pl-pds">'</span></span> <span class="pl-k">as</span> content, <span class="pl-s"><span class="pl-pds">'</span>x.jpg<span class="pl-pds">'</span></span> <span class="pl-k">as</span> content_filename</pre></div>
<p>This was while writing a unit test for <code>datasette-media</code> - for <a href="https://github.com/simonw/datasette-media/issues/19">issue #19</a>. I used it in the test <a href="https://github.com/simonw/datasette-media/blob/2cf64d949ccb8cd5f34b24aeb41b2a91de14cdd2/tests/test_media.py#L292-L295">here</a>.</p>
<p>The SQLite documentation for <a href="https://www.sqlite.org/lang_expr.html#literal_values_constants_" rel="nofollow">Literal values</a> explains how to do this:</p>
<blockquote>
<p>BLOB literals are string literals containing hexadecimal data and preceded by a single "x" or "X" character. Example: X'53514C697465'</p>
</blockquote>
<p>In Python 3 you can generate the hexadecimal representation of any byte string using <code>b'...'.hex()</code></p>
<p>So my solution looked like this:</p>
<div class="highlight highlight-source-python"><pre><span class="pl-s1">jpeg_bytes</span> <span class="pl-c1">=</span> <span class="pl-en">open</span>(<span class="pl-s">"content.jpg"</span>, <span class="pl-s">"rb"</span>).<span class="pl-en">read</span>()
<span class="pl-s1">sql</span> <span class="pl-c1">=</span> <span class="pl-s">"select X'{}' as content, 'x.jpg' as content_filename"</span>.<span class="pl-en">format</span>(<span class="pl-s1">jpeg_bytes</span>.<span class="pl-en">hex</span>())</pre></div>


path: sqlite_ld-preload.md
topic: sqlite
title: Using LD_PRELOAD to run any version of SQLite with Python
url: https://github.com/simonw/til/blob/main/sqlite/ld-preload.md
created: 2020-06-17T13:12:13-07:00
created_utc: 2020-06-17T20:12:13+00:00
updated: 2022-03-05T11:55:51-08:00
updated_utc: 2022-03-05T19:55:51+00:00
shot_hash: 794ccd0a4e60717c397ba7c2d9379330
slug: ld-preload
body:
I've been wanting to figure this out for ages. [This thread](https://news.ycombinator.com/item?id=23282684) on Hacker News plus [this Stackoverflow post](https://unix.stackexchange.com/questions/24811/changing-linked-library-for-a-given-executable-centos-6/24833#24833:) gave me some initial clues.

The challenge: someone reports a bug in Datasette with a specific version of SQLite. How can I run Datasette against that specific version?

Steps:

1. Download the source code for that specific version
2. Build a `libsqlite3.so` file using that source code
3. Run Datasette prefixed with `LD_PRELOAD=path/to/libsqlite3.so`

## Obtaining source code for old SQLite versions

This took a while to figure out. https://www.sqlite.org/chronology.html has a list of links to different releases. From there clicking on one of the dates takes you to a page like this: https://www.sqlite.org/src/timeline?c=cf538e2783&y=ci

From there clicking the "version-X" link takes you through to the timeline page like this: https://www.sqlite.org/src/timeline?r=version-3.8.11.1 - then click on e.g. "check-in: cf538e27" to get to a page like https://www.sqlite.org/src/info/cf538e2783e468bb which has a link to a Tarball and a Zip archive. Download one of those!

Hopefully shorter version: decide which version you want, then navigate to https://www.sqlite.org/src/timeline?t=version-3.8.11.1 and click the check-in link to get to the page that links to the Zip / Tarball.

Here's the URL to the 3.8.11.1 Tarball: https://www.sqlite.org/src/tarball/cf538e27/SQLite-cf538e27.tar.gz

## Building the library

I first tried this on macOS but I couldn't get the `LD_PRELOAD` trick to work. I think this is because of "system integrity protection" (clue in [this comment](https://github.com/nteract/nteract/issues/1523#issuecomment-284027093)) - so I decided to do it in a Docker Ubuntu container instead.

To start a fresh Ubuntu container:

    docker run -it -p 8001:8001 ubuntu

I'm using `-p 8001:8001` here to allow me to access Datasette running on port 8001 later on.

Now we need python3 and the various build tools (SQLite also needs `tcl` for the build):

    apt-get install -y python3 build-essential tcl wget

Annoyingly installing `tcl` requires us to interactively set the timezone data - I enter `2` for America and then `85` for Los Angeles.

Download and extract the SQLite code:

    cd /tmp
    wget https://www.sqlite.org/src/tarball/cf538e27/SQLite-cf538e27.tar.gz
    tar -xzvf SQLite-cf538e27.tar.gz
    cd SQLite-cf538e27

Now we can build the extension. The `CPPFLAGS` are optional but I found I needed them to get the full Datasette test suite to pass later on:

    CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1" ./configure
    make

There is no need to run `make install` here, we just need the compiled library.

This puts the libraries in a `.libs` directory:

    root@11b79085483e:/tmp/SQLite-cf538e27# ls -l .libs/
    total 22660
    -rw-r--r-- 1 root root 9622164 Jun 17 12:52 libsqlite3.a
    lrwxrwxrwx 1 root root      16 Jun 17 12:52 libsqlite3.la -> ../libsqlite3.la
    -rw-r--r-- 1 root root     955 Jun 17 12:52 libsqlite3.lai
    lrwxrwxrwx 1 root root      19 Jun 17 12:52 libsqlite3.so -> libsqlite3.so.0.8.6
    lrwxrwxrwx 1 root root      19 Jun 17 12:52 libsqlite3.so.0 -> libsqlite3.so.0.8.6
    -rwxr-xr-x 1 root root 4021816 Jun 17 12:52 libsqlite3.so.0.8.6
    -rwxr-xr-x 1 root root  314160 Jun 17 12:52 sqlite3
    -rw-r--r-- 1 root root 9232752 Jun 17 12:52 sqlite3.o

We can confirm that this worked using the following one-liner:

    LD_PRELOAD=.libs/libsqlite3.so python3 -c \
        'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'

Which for me now outputs:

    ('3.8.11.1',)

Try that without the `LD_PRELOAD` to see the difference:

    root@11b79085483e:/tmp/SQLite-cf538e27# python3 -c \
    > 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'
    ('3.31.1',)

## Running Datasette

No need for a virtual environent since we are in a Docker container. We need to install `pip`:

    apt-get install -y python3-pip

Then we can install Datasette:

    python3 -mpip install datasette

Now run it like this:

    datasette -h 0.0.0.0
    INFO:     Started server process [10266]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

We need `-h 0.0.0.0` to allow access from outside. On our machine we can now visit it here:

http://0.0.0.0:8001/-/versions

And confirm that the default SQLite version is `"version": "3.31.1"`.

Alternatively, run this to see the versions output without needing to run the server:

    datasette --get /-/versions.json

Quit Datasette and start it again with the `LD_PRELOAD`:

    LD_PRELOAD=.libs/libsqlite3.so datasette -h 0.0.0.0
    INFO:     Started server process [10274]
    INFO:     Waiting for application startup.
    INFO:     Application startup complete.
    INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)

And now http://0.0.0.0:8001/-/versions reports the following:

```json
{
    "python": {
        "version": "3.8.2",
        "full": "3.8.2 (default, Apr 27 2020, 15:53:34) \n[GCC 9.3.0]"
    },
    "datasette": {
        "version": "0.44"
    },
    "asgi": "3.0",
    "uvicorn": "0.11.5",
    "sqlite": {
        "version": "3.8.11.1",
        "fts_versions": [],
        "extensions": {},
        "compile_options": [
            "HAVE_ISNAN",
            "SYSTEM_MALLOC",
            "TEMP_STORE=1",
            "THREADSAFE=1"
        ]
    }
}
```

## Running the Datasette tests

To run Datasette's test suite I needed to install a few extra dependencies:

```
apt-get install -y python3-pip git python3.8-venv
cd /tmp
git clone https://github.com/simonw/datasette
cd datasette
python3 -m venv venv
source venv/bin/activate
pip install wheel # So bdist_wheel works in next step
pip install -e '.[test]'
LD_PRELOAD=/tmp/SQLite-cf538e27/.libs/libsqlite3.so python3 -c 
```
I used this process to track down a bug that was introduced between SQLite 3.36.0 and 3.37.0 in [this issue](https://github.com/simonw/datasette/issues/1647).

html:
<p>I've been wanting to figure this out for ages. <a href="https://news.ycombinator.com/item?id=23282684" rel="nofollow">This thread</a> on Hacker News plus <a href="https://unix.stackexchange.com/questions/24811/changing-linked-library-for-a-given-executable-centos-6/24833#24833:" rel="nofollow">this Stackoverflow post</a> gave me some initial clues.</p>
<p>The challenge: someone reports a bug in Datasette with a specific version of SQLite. How can I run Datasette against that specific version?</p>
<p>Steps:</p>
<ol>
<li>Download the source code for that specific version</li>
<li>Build a <code>libsqlite3.so</code> file using that source code</li>
<li>Run Datasette prefixed with <code>LD_PRELOAD=path/to/libsqlite3.so</code>
</li>
</ol>
<h2>
<a id="user-content-obtaining-source-code-for-old-sqlite-versions" class="anchor" href="#obtaining-source-code-for-old-sqlite-versions" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Obtaining source code for old SQLite versions</h2>
<p>This took a while to figure out. <a href="https://www.sqlite.org/chronology.html" rel="nofollow">https://www.sqlite.org/chronology.html</a> has a list of links to different releases. From there clicking on one of the dates takes you to a page like this: <a href="https://www.sqlite.org/src/timeline?c=cf538e2783&amp;y=ci" rel="nofollow">https://www.sqlite.org/src/timeline?c=cf538e2783&amp;y=ci</a></p>
<p>From there clicking the "version-X" link takes you through to the timeline page like this: <a href="https://www.sqlite.org/src/timeline?r=version-3.8.11.1" rel="nofollow">https://www.sqlite.org/src/timeline?r=version-3.8.11.1</a> - then click on e.g. "check-in: cf538e27" to get to a page like <a href="https://www.sqlite.org/src/info/cf538e2783e468bb" rel="nofollow">https://www.sqlite.org/src/info/cf538e2783e468bb</a> which has a link to a Tarball and a Zip archive. Download one of those!</p>
<p>Hopefully shorter version: decide which version you want, then navigate to <a href="https://www.sqlite.org/src/timeline?t=version-3.8.11.1" rel="nofollow">https://www.sqlite.org/src/timeline?t=version-3.8.11.1</a> and click the check-in link to get to the page that links to the Zip / Tarball.</p>
<p>Here's the URL to the 3.8.11.1 Tarball: <a href="https://www.sqlite.org/src/tarball/cf538e27/SQLite-cf538e27.tar.gz" rel="nofollow">https://www.sqlite.org/src/tarball/cf538e27/SQLite-cf538e27.tar.gz</a></p>
<h2>
<a id="user-content-building-the-library" class="anchor" href="#building-the-library" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Building the library</h2>
<p>I first tried this on macOS but I couldn't get the <code>LD_PRELOAD</code> trick to work. I think this is because of "system integrity protection" (clue in <a href="https://github.com/nteract/nteract/issues/1523#issuecomment-284027093">this comment</a>) - so I decided to do it in a Docker Ubuntu container instead.</p>
<p>To start a fresh Ubuntu container:</p>
<pre><code>docker run -it -p 8001:8001 ubuntu
</code></pre>
<p>I'm using <code>-p 8001:8001</code> here to allow me to access Datasette running on port 8001 later on.</p>
<p>Now we need python3 and the various build tools (SQLite also needs <code>tcl</code> for the build):</p>
<pre><code>apt-get install -y python3 build-essential tcl wget
</code></pre>
<p>Annoyingly installing <code>tcl</code> requires us to interactively set the timezone data - I enter <code>2</code> for America and then <code>85</code> for Los Angeles.</p>
<p>Download and extract the SQLite code:</p>
<pre><code>cd /tmp
wget https://www.sqlite.org/src/tarball/cf538e27/SQLite-cf538e27.tar.gz
tar -xzvf SQLite-cf538e27.tar.gz
cd SQLite-cf538e27
</code></pre>
<p>Now we can build the extension. The <code>CPPFLAGS</code> are optional but I found I needed them to get the full Datasette test suite to pass later on:</p>
<pre><code>CPPFLAGS="-DSQLITE_ENABLE_FTS3 -DSQLITE_ENABLE_FTS3_PARENTHESIS -DSQLITE_ENABLE_RTREE=1" ./configure
make
</code></pre>
<p>There is no need to run <code>make install</code> here, we just need the compiled library.</p>
<p>This puts the libraries in a <code>.libs</code> directory:</p>
<pre><code>root@11b79085483e:/tmp/SQLite-cf538e27# ls -l .libs/
total 22660
-rw-r--r-- 1 root root 9622164 Jun 17 12:52 libsqlite3.a
lrwxrwxrwx 1 root root      16 Jun 17 12:52 libsqlite3.la -&gt; ../libsqlite3.la
-rw-r--r-- 1 root root     955 Jun 17 12:52 libsqlite3.lai
lrwxrwxrwx 1 root root      19 Jun 17 12:52 libsqlite3.so -&gt; libsqlite3.so.0.8.6
lrwxrwxrwx 1 root root      19 Jun 17 12:52 libsqlite3.so.0 -&gt; libsqlite3.so.0.8.6
-rwxr-xr-x 1 root root 4021816 Jun 17 12:52 libsqlite3.so.0.8.6
-rwxr-xr-x 1 root root  314160 Jun 17 12:52 sqlite3
-rw-r--r-- 1 root root 9232752 Jun 17 12:52 sqlite3.o
</code></pre>
<p>We can confirm that this worked using the following one-liner:</p>
<pre><code>LD_PRELOAD=.libs/libsqlite3.so python3 -c \
    'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'
</code></pre>
<p>Which for me now outputs:</p>
<pre><code>('3.8.11.1',)
</code></pre>
<p>Try that without the <code>LD_PRELOAD</code> to see the difference:</p>
<pre><code>root@11b79085483e:/tmp/SQLite-cf538e27# python3 -c \
&gt; 'import sqlite3; print(sqlite3.connect(":memory").execute("select sqlite_version()").fetchone())'
('3.31.1',)
</code></pre>
<h2>
<a id="user-content-running-datasette" class="anchor" href="#running-datasette" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running Datasette</h2>
<p>No need for a virtual environent since we are in a Docker container. We need to install <code>pip</code>:</p>
<pre><code>apt-get install -y python3-pip
</code></pre>
<p>Then we can install Datasette:</p>
<pre><code>python3 -mpip install datasette
</code></pre>
<p>Now run it like this:</p>
<pre><code>datasette -h 0.0.0.0
INFO:     Started server process [10266]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
</code></pre>
<p>We need <code>-h 0.0.0.0</code> to allow access from outside. On our machine we can now visit it here:</p>
<p><a href="http://0.0.0.0:8001/-/versions" rel="nofollow">http://0.0.0.0:8001/-/versions</a></p>
<p>And confirm that the default SQLite version is <code>"version": "3.31.1"</code>.</p>
<p>Alternatively, run this to see the versions output without needing to run the server:</p>
<pre><code>datasette --get /-/versions.json
</code></pre>
<p>Quit Datasette and start it again with the <code>LD_PRELOAD</code>:</p>
<pre><code>LD_PRELOAD=.libs/libsqlite3.so datasette -h 0.0.0.0
INFO:     Started server process [10274]
INFO:     Waiting for application startup.
INFO:     Application startup complete.
INFO:     Uvicorn running on http://0.0.0.0:8001 (Press CTRL+C to quit)
</code></pre>
<p>And now <a href="http://0.0.0.0:8001/-/versions" rel="nofollow">http://0.0.0.0:8001/-/versions</a> reports the following:</p>
<div class="highlight highlight-source-json"><pre>{
    <span class="pl-ent">"python"</span>: {
        <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.8.2<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"full"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.8.2 (default, Apr 27 2020, 15:53:34) <span class="pl-cce">\n</span>[GCC 9.3.0]<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"datasette"</span>: {
        <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.44<span class="pl-pds">"</span></span>
    },
    <span class="pl-ent">"asgi"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.0<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"uvicorn"</span>: <span class="pl-s"><span class="pl-pds">"</span>0.11.5<span class="pl-pds">"</span></span>,
    <span class="pl-ent">"sqlite"</span>: {
        <span class="pl-ent">"version"</span>: <span class="pl-s"><span class="pl-pds">"</span>3.8.11.1<span class="pl-pds">"</span></span>,
        <span class="pl-ent">"fts_versions"</span>: [],
        <span class="pl-ent">"extensions"</span>: {},
        <span class="pl-ent">"compile_options"</span>: [
            <span class="pl-s"><span class="pl-pds">"</span>HAVE_ISNAN<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>SYSTEM_MALLOC<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>TEMP_STORE=1<span class="pl-pds">"</span></span>,
            <span class="pl-s"><span class="pl-pds">"</span>THREADSAFE=1<span class="pl-pds">"</span></span>
        ]
    }
}</pre></div>
<h2>
<a id="user-content-running-the-datasette-tests" class="anchor" href="#running-the-datasette-tests" aria-hidden="true"><span aria-hidden="true" class="octicon octicon-link"></span></a>Running the Datasette tests</h2>
<p>To run Datasette's test suite I needed to install a few extra dependencies:</p>
<pre><code>apt-get install -y python3-pip git python3.8-venv
cd /tmp
git clone https://github.com/simonw/datasette
cd datasette
python3 -m venv venv
source venv/bin/activate
pip install wheel # So bdist_wheel works in next step
pip install -e '.[test]'
LD_PRELOAD=/tmp/SQLite-cf538e27/.libs/libsqlite3.so python3 -c 
</code></pre>
<p>I used this process to track down a bug that was introduced between SQLite 3.36.0 and 3.37.0 in <a href="https://github.com/simonw/datasette/issues/1647">this issue</a>.</p>


