til: fly_fly-logs-to-s3.md

This data as json

path topic title url body html shot created created_utc updated updated_utc shot_hash slug
fly_fly-logs-to-s3.md fly Writing Fly logs to S3 https://github.com/simonw/til/blob/main/fly/fly-logs-to-s3.md [Fly](https://fly.io/) offers [fly-log-shipper](https://github.com/superfly/fly-log-shipper) as a container you can run in a Fly application to send all of the logs from your other applications to a logging provider. Several providers are supported. I decided to write them to an S3 bucket. ## Bucket credentials I used my [s3-credentials](https://github.com/simonw/s3-credentials) tool to generate an access key and secret locked down to just one newly created bucket: ``` s3-credentials create my-project-fly-logs \ --format ini \ --bucket-region us-west-1 \ --create-bucket \ > logging-credentials.txt ``` I chose `us-west-1` or Northern California as the region, as it is closest to me. That command output the following: ``` Created bucket: my-project-fly-logs in region: us-west-1 Created user: 's3.read-write.my-project-fly-logs' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess' Attached policy s3.read-write.my-project-fly-logs to user s3.read-write.my-project-fly-logs Created access key for user: s3.read-write.my-project-fly-logs ``` The full set of configuration needed by `fly-log-shipper` for S3 is: - `ORG` - Fly organisation slug - `ACCESS_TOKEN` - Fly personal access token - `AWS_ACCESS_KEY_ID` - AWS Access key with access to the log bucket - `AWS_SECRET_ACCESS_KEY` - AWS secret access key - `AWS_BUCKET` - AWS S3 bucket to store logs in - `AWS_REGION` - Region for the bucket I created a personal access token at https://fly.io/user/personal_access_tokens ## Creating the app I created a new Fly application to run the container like so: fly apps create --name my-project-log-shipper --org my-project-org ## Setting the secrets I set all of the configuration variables as secrets in one go like this: ``` fly secrets set \ ORG="my-project-org" \ ACCESS_TOKEN="..." \ AWS_ACCESS_KEY_ID="AKIAWXFXAIOZIPTTHMBQ" \ AWS_SECRET_ACCESS_KEY="..." \ AWS_BUCKET="my-project-fly-logs" \ AWS_REGION="us-west-1" \ -a my-project-shipper ``` ## Deploying the app It turns out you still need a `fly.toml` file to successfully deploy - I tried without and got healthcheck errors. Here's the TOML file I used: ```toml app = "my-project-log-shipper" [metrics] port = 9598 path = "/metrics" ``` Then I passed that to the `fly deploy` command like this: ``` fly deploy --image ghcr.io/superfly/fly-log-shipper:v0.0.1 -a my-project-log-shipper -c fly.toml ``` Output: ``` ==> Verifying app config --> Verified app config ==> Building image Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' locally... Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' remotely... image found: img_19gm46rqwn14x0jk ==> Creating release --> release v1 created --> You can detach the terminal anytime without stopping the deployment ==> Monitoring deployment 1 desired, 1 placed, 1 healthy, 0 unhealthy --> v1 deployed successfully ``` I used [Transmit](https://panic.com/transmit/) to inspect the bucket and my logs are showing up there as intended. I can also now list my log files using `s3-credentials list-bucket` and the credentials I saved earlier: ``` % s3-credentials list-bucket -a logging-credentials.txt my-project-fly-logs --csv Key,LastModified,ETag,Size,StorageClass,Owner my-project-admin/2022-05-25//1653507279-b9534d49-546a-47cb-bd93-36e08b8457ee.log.gz,2022-05-25 19:34:40+00:00,"""8b499f523bd2f5e9438cf2c0d42eab8c""",335,STANDARD, my-project-admin/2022-05-25//1653507625-fc3c0f32-b3e8-4681-98ce-a4fe2960d1b2.log.gz,2022-05-25 19:40:26+00:00,"""5fe1f6fbf747706930edcfd00e3dd4fe""",444,STANDARD, my-project-admin/2022-05-25//1653508693-7f4da568-e462-46a2-9904-e3ac801c7fee.log.gz,2022-05-25 19:58:14+00:00,"""794c57787a692a17d1fbfb78df442d58""",353,STANDARD, my-project-postgresql/2022-05-25//1653507222-60e072e2-fd78-437f-a097-c063df424224.log.gz,2022-05-25 19:33:43+00:00,"""5d9965b79260e6a0f27cff5f02eee71d""",3376,STANDARD, my-project-postgresql/2022-05-25//1653507525-b3a04523-8732-4072-92c0-57dd06a680b1.log.gz,2022-05-25 ``` **Update:** My initial attempt at doing this had a small bug: those `//` folder separators in that output turn out [to break AWS Athena](https://til.simonwillison.net/aws/athena-key-does-not-exist). Deploying the latest version of the `fly-log-shipper` directly from their repository seems to fix that though (with [this fix](https://github.com/superfly/fly-log-shipper/pull/29) applied too). <p><a href="https://fly.io/" rel="nofollow">Fly</a> offers <a href="https://github.com/superfly/fly-log-shipper">fly-log-shipper</a> as a container you can run in a Fly application to send all of the logs from your other applications to a logging provider.</p> <p>Several providers are supported. I decided to write them to an S3 bucket.</p> <h2><a id="user-content-bucket-credentials" class="anchor" aria-hidden="true" href="#bucket-credentials"><span aria-hidden="true" class="octicon octicon-link"></span></a>Bucket credentials</h2> <p>I used my <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> tool to generate an access key and secret locked down to just one newly created bucket:</p> <pre><code>s3-credentials create my-project-fly-logs \ --format ini \ --bucket-region us-west-1 \ --create-bucket \ &gt; logging-credentials.txt </code></pre> <p>I chose <code>us-west-1</code> or Northern California as the region, as it is closest to me.</p> <p>That command output the following:</p> <pre><code>Created bucket: my-project-fly-logs in region: us-west-1 Created user: 's3.read-write.my-project-fly-logs' with permissions boundary: 'arn:aws:iam::aws:policy/AmazonS3FullAccess' Attached policy s3.read-write.my-project-fly-logs to user s3.read-write.my-project-fly-logs Created access key for user: s3.read-write.my-project-fly-logs </code></pre> <p>The full set of configuration needed by <code>fly-log-shipper</code> for S3 is:</p> <ul> <li> <code>ORG</code> - Fly organisation slug</li> <li> <code>ACCESS_TOKEN</code> - Fly personal access token</li> <li> <code>AWS_ACCESS_KEY_ID</code> - AWS Access key with access to the log bucket</li> <li> <code>AWS_SECRET_ACCESS_KEY</code> - AWS secret access key</li> <li> <code>AWS_BUCKET</code> - AWS S3 bucket to store logs in</li> <li> <code>AWS_REGION</code> - Region for the bucket</li> </ul> <p>I created a personal access token at <a href="https://fly.io/user/personal_access_tokens" rel="nofollow">https://fly.io/user/personal_access_tokens</a></p> <h2><a id="user-content-creating-the-app" class="anchor" aria-hidden="true" href="#creating-the-app"><span aria-hidden="true" class="octicon octicon-link"></span></a>Creating the app</h2> <p>I created a new Fly application to run the container like so:</p> <pre><code>fly apps create --name my-project-log-shipper --org my-project-org </code></pre> <h2><a id="user-content-setting-the-secrets" class="anchor" aria-hidden="true" href="#setting-the-secrets"><span aria-hidden="true" class="octicon octicon-link"></span></a>Setting the secrets</h2> <p>I set all of the configuration variables as secrets in one go like this:</p> <pre><code>fly secrets set \ ORG="my-project-org" \ ACCESS_TOKEN="..." \ AWS_ACCESS_KEY_ID="AKIAWXFXAIOZIPTTHMBQ" \ AWS_SECRET_ACCESS_KEY="..." \ AWS_BUCKET="my-project-fly-logs" \ AWS_REGION="us-west-1" \ -a my-project-shipper </code></pre> <h2><a id="user-content-deploying-the-app" class="anchor" aria-hidden="true" href="#deploying-the-app"><span aria-hidden="true" class="octicon octicon-link"></span></a>Deploying the app</h2> <p>It turns out you still need a <code>fly.toml</code> file to successfully deploy - I tried without and got healthcheck errors.</p> <p>Here's the TOML file I used:</p> <div class="highlight highlight-source-toml"><pre><span class="pl-smi">app</span> = <span class="pl-s"><span class="pl-pds">"</span>my-project-log-shipper<span class="pl-pds">"</span></span> [<span class="pl-en">metrics</span>] <span class="pl-smi">port</span> = <span class="pl-c1">9598</span> <span class="pl-smi">path</span> = <span class="pl-s"><span class="pl-pds">"</span>/metrics<span class="pl-pds">"</span></span></pre></div> <p>Then I passed that to the <code>fly deploy</code> command like this:</p> <pre><code>fly deploy --image ghcr.io/superfly/fly-log-shipper:v0.0.1 -a my-project-log-shipper -c fly.toml </code></pre> <p>Output:</p> <pre><code>==&gt; Verifying app config --&gt; Verified app config ==&gt; Building image Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' locally... Searching for image 'ghcr.io/superfly/fly-log-shipper:v0.0.1' remotely... image found: img_19gm46rqwn14x0jk ==&gt; Creating release --&gt; release v1 created --&gt; You can detach the terminal anytime without stopping the deployment ==&gt; Monitoring deployment 1 desired, 1 placed, 1 healthy, 0 unhealthy --&gt; v1 deployed successfully </code></pre> <p>I used <a href="https://panic.com/transmit/" rel="nofollow">Transmit</a> to inspect the bucket and my logs are showing up there as intended.</p> <p>I can also now list my log files using <code>s3-credentials list-bucket</code> and the credentials I saved earlier:</p> <pre><code>% s3-credentials list-bucket -a logging-credentials.txt my-project-fly-logs --csv Key,LastModified,ETag,Size,StorageClass,Owner my-project-admin/2022-05-25//1653507279-b9534d49-546a-47cb-bd93-36e08b8457ee.log.gz,2022-05-25 19:34:40+00:00,"""8b499f523bd2f5e9438cf2c0d42eab8c""",335,STANDARD, my-project-admin/2022-05-25//1653507625-fc3c0f32-b3e8-4681-98ce-a4fe2960d1b2.log.gz,2022-05-25 19:40:26+00:00,"""5fe1f6fbf747706930edcfd00e3dd4fe""",444,STANDARD, my-project-admin/2022-05-25//1653508693-7f4da568-e462-46a2-9904-e3ac801c7fee.log.gz,2022-05-25 19:58:14+00:00,"""794c57787a692a17d1fbfb78df442d58""",353,STANDARD, my-project-postgresql/2022-05-25//1653507222-60e072e2-fd78-437f-a097-c063df424224.log.gz,2022-05-25 19:33:43+00:00,"""5d9965b79260e6a0f27cff5f02eee71d""",3376,STANDARD, my-project-postgresql/2022-05-25//1653507525-b3a04523-8732-4072-92c0-57dd06a680b1.log.gz,2022-05-25 </code></pre> <p><strong>Update:</strong> My initial attempt at doing this had a small bug: those <code>//</code> folder separators in that output turn out <a href="https://til.simonwillison.net/aws/athena-key-does-not-exist" rel="nofollow">to break AWS Athena</a>.</p> <p>Deploying the latest version of the <code>fly-log-shipper</code> directly from their repository seems to fix that though (with <a href="https://github.com/superfly/fly-log-shipper/pull/29">this fix</a> applied too).</p> <Binary: 55,413 bytes> 2022-05-25T12:47:40-07:00 2022-05-25T19:47:40+00:00 2022-09-27T20:59:18-07:00 2022-09-28T03:59:18+00:00 3ac2c543a1de8af7c9207aa5896ba423 fly-logs-to-s3