til: bash_use-awk-to-add-a-prefix.md
This data as json
| path | topic | title | url | body | html | shot | created | created_utc | updated | updated_utc | shot_hash | slug |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| bash_use-awk-to-add-a-prefix.md | bash | Using awk to add a prefix | https://github.com/simonw/til/blob/main/bash/use-awk-to-add-a-prefix.md | I wanted to dynamically run the following command against all files in a directory: ```bash pypi-to-sqlite content.db -f /tmp/pypi-datasette-packages/packages/airtable-export.json \ -f /tmp/pypi-datasette-packages/packages/csv-diff.json \ --prefix pypi_ ``` I can't use `/tmp/pypi-datasette-packages/packages/*.json` here because I need each file to be processed using the `-f` option. I found a solution using `awk`. The `awk` program `'{print "-f "$0}'` adds a prefix to the input, for example: ``` % echo "blah" | awk '{print "-f "$0}' -f blah ``` I wanted that trailing backslash too, so I used this: ```awk {print "-f "$0 " \\"} ``` Piping to `awk` works, so I combined that with `ls ../*.json` like so: ``` % ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}' -f /tmp/pypi-datasette-packages/packages/airtable-export.json \ -f /tmp/pypi-datasette-packages/packages/csv-diff.json \ -f /tmp/pypi-datasette-packages/packages/csvs-to-sqlite.json \ ``` Then I used `eval` to execute the command. The full recipe looks like this: ```bash args=$(ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}') eval "pypi-to-sqlite content.db $args --prefix pypi_" ``` Full details in [datasette.io issue 98](https://github.com/simonw/datasette.io/issues/98). | <p>I wanted to dynamically run the following command against all files in a directory:</p> <div class="highlight highlight-source-shell"><pre>pypi-to-sqlite content.db -f /tmp/pypi-datasette-packages/packages/airtable-export.json \ -f /tmp/pypi-datasette-packages/packages/csv-diff.json \ --prefix pypi_</pre></div> <p>I can't use <code>/tmp/pypi-datasette-packages/packages/*.json</code> here because I need each file to be processed using the <code>-f</code> option.</p> <p>I found a solution using <code>awk</code>. The <code>awk</code> program <code>'{print "-f "$0}'</code> adds a prefix to the input, for example:</p> <pre><code>% echo "blah" | awk '{print "-f "$0}' -f blah </code></pre> <p>I wanted that trailing backslash too, so I used this:</p> <div class="highlight highlight-source-awk"><pre>{<span class="pl-k">print</span> <span class="pl-s"><span class="pl-pds">"</span>-f <span class="pl-pds">"</span></span><span class="pl-c1">$0</span> <span class="pl-s"><span class="pl-pds">"</span> <span class="pl-cce">\\</span><span class="pl-pds">"</span></span>}</pre></div> <p>Piping to <code>awk</code> works, so I combined that with <code>ls ../*.json</code> like so:</p> <pre><code>% ls /tmp/pypi-datasette-packages/packages/*.json | awk '{print "-f "$0 " \\"}' -f /tmp/pypi-datasette-packages/packages/airtable-export.json \ -f /tmp/pypi-datasette-packages/packages/csv-diff.json \ -f /tmp/pypi-datasette-packages/packages/csvs-to-sqlite.json \ </code></pre> <p>Then I used <code>eval</code> to execute the command. The full recipe looks like this:</p> <div class="highlight highlight-source-shell"><pre>args=<span class="pl-s"><span class="pl-pds">$(</span>ls /tmp/pypi-datasette-packages/packages/<span class="pl-k">*</span>.json <span class="pl-k">|</span> awk <span class="pl-s"><span class="pl-pds">'</span>{print "-f "$0 " \\"}<span class="pl-pds">'</span></span><span class="pl-pds">)</span></span> <span class="pl-c1">eval</span> <span class="pl-s"><span class="pl-pds">"</span>pypi-to-sqlite content.db <span class="pl-smi">$args</span></span> <span class="pl-s">--prefix pypi_<span class="pl-pds">"</span></span></pre></div> <p>Full details in <a href="https://github.com/simonw/datasette.io/issues/98">datasette.io issue 98</a>.</p> | <Binary: 61,224 bytes> | 2022-04-08T09:25:04-07:00 | 2022-04-08T16:25:04+00:00 | 2022-04-08T09:25:04-07:00 | 2022-04-08T16:25:04+00:00 | 801ca3f33198f55a114494b5608cb6c1 | use-awk-to-add-a-prefix |