til: aws_helper-for-boto-aws-pagination.md
This data as json
| path | topic | title | url | body | html | shot | created | created_utc | updated | updated_utc | shot_hash | slug |
|---|---|---|---|---|---|---|---|---|---|---|---|---|
| aws_helper-for-boto-aws-pagination.md | aws | Helper function for pagination using AWS boto3 | https://github.com/simonw/til/blob/main/aws/helper-for-boto-aws-pagination.md | I noticed that a lot of my boto3 code in [s3-credentials](https://github.com/simonw/s3-credentials) looked like this: ```python paginator = iam.get_paginator("list_user_policies") for response in paginator.paginate(UserName=username): for policy_name in response["PolicyNames"]: print(policy_name) ``` This was enough verbosity that I was hesitating on implementing pagination properly for some method calls. I came up with this helper function to use instead: ```python def paginate(service, method, list_key, **kwargs): paginator = service.get_paginator(method) for response in paginator.paginate(**kwargs): yield from response[list_key] ``` Now the above becomes: ```python for policy_name in paginate(iam, "list_user_policies", "PolicyNames", UserName=username): print(policy_name) ``` Here's [the issue](https://github.com/simonw/s3-credentials/issues/63) and the [refactoring commit](https://github.com/simonw/s3-credentials/commit/fc1e06ca3ffa2c73e196cffe741ef4e950204240). | <p>I noticed that a lot of my boto3 code in <a href="https://github.com/simonw/s3-credentials">s3-credentials</a> looked like this:</p> <div class="highlight highlight-source-python"><pre><span class="pl-s1">paginator</span> <span class="pl-c1">=</span> <span class="pl-s1">iam</span>.<span class="pl-en">get_paginator</span>(<span class="pl-s">"list_user_policies"</span>) <span class="pl-k">for</span> <span class="pl-s1">response</span> <span class="pl-c1">in</span> <span class="pl-s1">paginator</span>.<span class="pl-en">paginate</span>(<span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s1">username</span>): <span class="pl-k">for</span> <span class="pl-s1">policy_name</span> <span class="pl-c1">in</span> <span class="pl-s1">response</span>[<span class="pl-s">"PolicyNames"</span>]: <span class="pl-en">print</span>(<span class="pl-s1">policy_name</span>)</pre></div> <p>This was enough verbosity that I was hesitating on implementing pagination properly for some method calls.</p> <p>I came up with this helper function to use instead:</p> <div class="highlight highlight-source-python"><pre><span class="pl-k">def</span> <span class="pl-en">paginate</span>(<span class="pl-s1">service</span>, <span class="pl-s1">method</span>, <span class="pl-s1">list_key</span>, <span class="pl-c1">**</span><span class="pl-s1">kwargs</span>): <span class="pl-s1">paginator</span> <span class="pl-c1">=</span> <span class="pl-s1">service</span>.<span class="pl-en">get_paginator</span>(<span class="pl-s1">method</span>) <span class="pl-k">for</span> <span class="pl-s1">response</span> <span class="pl-c1">in</span> <span class="pl-s1">paginator</span>.<span class="pl-en">paginate</span>(<span class="pl-c1">**</span><span class="pl-s1">kwargs</span>): <span class="pl-k">yield</span> <span class="pl-k">from</span> <span class="pl-s1">response</span>[<span class="pl-s1">list_key</span>]</pre></div> <p>Now the above becomes:</p> <div class="highlight highlight-source-python"><pre><span class="pl-k">for</span> <span class="pl-s1">policy_name</span> <span class="pl-c1">in</span> <span class="pl-en">paginate</span>(<span class="pl-s1">iam</span>, <span class="pl-s">"list_user_policies"</span>, <span class="pl-s">"PolicyNames"</span>, <span class="pl-v">UserName</span><span class="pl-c1">=</span><span class="pl-s1">username</span>): <span class="pl-en">print</span>(<span class="pl-s1">policy_name</span>)</pre></div> <p>Here's <a href="https://github.com/simonw/s3-credentials/issues/63">the issue</a> and the <a href="https://github.com/simonw/s3-credentials/commit/fc1e06ca3ffa2c73e196cffe741ef4e950204240">refactoring commit</a>.</p> | <Binary: 64,562 bytes> | 2022-01-19T11:49:58-08:00 | 2022-01-19T19:49:58+00:00 | 2022-01-19T11:49:58-08:00 | 2022-01-19T19:49:58+00:00 | c6247a16c6d08af6de0042edcc3e518d | helper-for-boto-aws-pagination |