I wanted to add an action option to the Django Admin for exporting the currently selected set of rows (or every row in the table) as a CSV file.
\nI ended up using a pattern inspired by this Django Snippet, but with an added touch for more efficient exports. In order to avoid using up too much memory for the export, I use keyset pagination to fetch 500 rows at a time.
\nThe keyset_pagination_iterator() helper function accepts any queryset, orders it by the primary key and then repeatedly fetches 500 items. It then modifies the queryset to add a WHERE id > $last_seen_id clause. This is a relatively inexpensive way to paginate, so having an endpoint perform that query dozens or even hundreds of times should hopefully avoid adding too much load to the database.
The action itself uses a pattern that combines StringIO and csv.writer() to stream out the results as a CSV file.
Django's StreamingHttpResponse mechanism is really neat: it accepts a Python iterator or generator and returns a streaming response derived from that sequence.
The Django documentation says \"Streaming responses will tie a worker process for the entire duration of the response. This may result in poor performance\" - this particular project runs on Google Cloud Run so I'm less concerned about tying up a worker than I would be normally, plus the export option is only available to trusted staff users with access to the Django Admin interface.
\nTo add the CSV export option to a ModelAdmin subclass, do the following:
from .admin_actions import export_as_csv_action\n\n@admin.register(County)\nclass CountyAdmin(admin.ModelAdmin):\n actions = [export_as_csv_action()]Here's admin_actions.py:
import csv\nfrom io import StringIO\n\nfrom django.http import StreamingHttpResponse\n\n\ndef keyset_pagination_iterator(input_queryset, batch_size=500):\n all_queryset = input_queryset.order_by(\"pk\")\n last_pk = None\n while True:\n queryset = all_queryset\n if last_pk is not None:\n queryset = all_queryset.filter(pk__gt=last_pk)\n queryset = queryset[:batch_size]\n for row in queryset:\n last_pk = row.pk\n yield row\n if not queryset:\n break\n\n\ndef export_as_csv_action(description=\"Export selected rows to CSV\"):\n def export_as_csv(modeladmin, request, queryset):\n def rows(queryset):\n\n csvfile = StringIO()\n csvwriter = csv.writer(csvfile)\n columns = [field.name for field in modeladmin.model._meta.fields]\n\n def read_and_flush():\n csvfile.seek(0)\n data = csvfile.read()\n csvfile.seek(0)\n csvfile.truncate()\n return data\n\n header = False\n\n if not header:\n header = True\n csvwriter.writerow(columns)\n yield read_and_flush()\n\n for row in keyset_pagination_iterator(queryset):\n csvwriter.writerow(getattr(row, column) for column in columns)\n yield read_and_flush()\n\n response = StreamingHttpResponse(rows(queryset), content_type=\"text/csv\")\n response[\"Content-Disposition\"] = (\n \"attachment; filename=%s.csv\" % modeladmin.model.__name__\n )\n\n return response\n\n export_as_csv.short_description = description\n return export_as_csv