Datasette News

I built a ChatGPT plugin to answer questions about data hosted in Datasette describes a new experimental Datasette plugin to enable people to query data hosted in a Datasette interface via ChatGPT, asking human language questions that are automatically converted to SQL and used to generate a readable response.

Using Datasette in GitHub Codespaces is a new tutorial showing how Datasette can be run in GitHub's free Codespaces browser-based development environments, using the new datasette-codespaces plugin.

Examples of sites built using Datasette now includes screenshots of Datasette deployments that illustrate a variety of problems that can be addressed using Datasette and its plugins.

Semantic search answers: Q&A against documentation with GPT3 + OpenAI embeddings shows how Datasette can be used to implement semantic search and build a system for answering questions against an existing corpus of text, using two new plugins: datasette-openai and datasette-faiss, and a new tool: openai-to-sqlite.

Datasette 0.64 is out, and includes a strong warning against running SpatiaLite in production without disabling arbitrary SQL queries, plus a new --setting default_allow_sql off setting to make it easier to do that. See Datasette 0.64, with a warning about SpatiaLite for more about this release. A new tutorial, Building a location to time zone API with SpatiaLite, describes how to safely use SpatiaLite and Datasette to build and deploy an API for looking up time zones for a latitude/longitude location.

Datasette 1.0a2: Upserts and finely grained permissions describes the new upsert API and much improved permissions capabilities introduced in the latest Datasette 1.0a2 alpha release.

Datasette’s new JSON write API: The first alpha of Datasette 1.0 introduces the new write API shipped in the first of the Datasette 1.0 alpha series of releases, including detailed descriptions of two demos that show how the API can be used.

Datasette 0.63 is out. Here are the annotated release notes.

Exploring the training data behind Stable Diffusion describes the process of building and deploying a 4GB searchable SQLite database using Datasette, starting with Parquet data that was used to train the Stable Diffusion image generation model. See also Exploring 12 Million of the 2.3 Billion Images Used to Train Stable Diffusion’s Image Generator.

Analyzing ScotRail audio announcements with Datasette—from prototype to production provides a detailed walk-through of the process of constructing an initial rapid prototype using Datasette Lite, extending it with a custom plugin and then deploying it as a full Datasette instance using GitHub Actions and Vercel.

Datasette 0.62 introduces compatibility with Pyodide for Datasette Lite, and incorporates a number of bug fixes, plugin hook upgrades and other improvements.

New tutorial and accompanying ten minute video: Cleaning data with sqlite-utils and Datasette.

s3-ocr is a new tool which can run OCR (via Amazon Textract) against every PDF file in an S3 bucket and write the results to a searchable SQLite database, ready to use with Datasette. Read more about it in s3-ocr: Extract text from PDF files stored in an S3 bucket.

Datasette Lite is a new way to run Datasette: entirely in your browser, thanks to the Pyodide project which provides a full Python environment compiled to WebAssembly. You can use it to explore any SQLite database file hosted on a CORS-enabled static hosting provider, which includes GitHub and GitHub Pages. Read more about this project in Datasette Lite: a server-side Python web application running in a browser.

Datasette for geospatial analysis describes how Datasette can be used in conjunction with SpatiaLite to work with geospatial data, including details of several geospatial plugins and tools from the Datasette ecosystem.

Datasette 0.61 introduces two potentially backwards-incompatible changes in preparation for the forthcoming 1.0 release: hashed URL mode has been moved to a new plugin, and the way URLs are generated to tables or databases containing special characters such as . or / has changed. Datasette 0.61.1 fixes a small bug in that release. See also the annotated release notes for these two versions.

The first two of an ongoing series of official Datasette tutorials are now available: Exploring a database with Datasette introduces the Datasette web interface and shows how it can be used to explore a new database, and Learn SQL with Datasette provides an introduction to SQL using Datasette as a learning environment.

Datasette 0.60 adds a new filters_from_request plugin hook, new internal methods for writing to the database, better performance and various faceting improvements. See also the annotated release notes.

Observable notebooks recently added a SQL cell type, allowing SQL queries to be executed as part of an interactive notebook workflow. Alex Garcia built a Datasette Client for these which allows you to excute queries against any Datasette instance and explore and visualize the results using JavaScript code running in a notebook.

Datasette 0.59 adds column descriptions in metadata, a new register_command plugin hook, enhanced --cors support and a bunch of other fixes and documentation improvements. See also the annotated release notes.

Datasette Desktop is a new macOS desktop application version of Datasette, which supports opening SQLite files on your computer, importing CSV files and installing plugins. I wrote more about how it works in Datasette Desktop—a macOS desktop application for Datasette.

The Baked Data architectural pattern describes a pattern commonly used with Datasette where the content for a site is bundled inside a SQLite database file and included alongside templates and application code in a deployment to a serverless hosting provider.

Datasette 0.58 has new plugin hooks, a huge performance improvement for faceting, support for Unix domain sockets and several other improvements. Read the annotated release notes for extra background and context on the release.

Datasette 0.57 is out with an important security patch plus a number of new features and bug fixes. Datasette 0.56.1, also out today, provides the security patch for users who are not yet ready to upgrade to the latest version.

Django SQL Dashboard is a new tool that brings a useful authenticated subset of Datasette to Django projects that are built on top of PostgreSQL.

Datasette 0.56 has bug fixes and documentation improvements, plus some new documented internal APIs for plugin authors and SpatiaLite 5 bundled with the official Datasette Docker container.

Datasette 0.55 adds support for cross-database SQL queries. You can now run datasette --crossdb one.db two.db and then run queries that join data from tables in both of those database files - see cross-database queries in the documentation for more details.

sqlite-utils 3.6 adds similar features: a db.attach(alias, filepath) Python API method and --attach alias filepath.db command-line option, both for attaching additional databases in order to execute cross-database queries.

This new Video introduction to Datasette and sqlite-utils provides a full introduction to both Datasette and sqlite-utils in 17 minutes, including a live demo of creating a database from a CSV file and publishing it to Google Cloud Run.

Serving map tiles from SQLite with MBTiles and datasette-tiles. datasette-tiles is a new plugin that adds a tile server to Datasette, serving map tiles from databases that conform to the MBTiles specification. download-tiles is a tool for building these databases, and datasette-basemap is a plugin that bundles a 22MB SQLite database with OpenStreetMap tiles covering zoom levels 0-6 for the entire world.

Datasette 0.54 is out today. Highlights include the new _internal in-memory database exposing details of connected tables, plus support for JavaScript modules in plugins and add-on scripts. More commentary on this release is available in the annotated release notes.

Drawing shapes on a map to query a SpatiaLite database introduces the new datasette-leaflet-freedraw plugin, which adds support for drawing shapes on a map to specify a GeoJSON MultiPolygon that can be used to query SpatiaLite databases.

APIs from CSS without JavaScript: the datasette-css-properties plugin introduces datasette-css-properties, a highly experimental plugin that can output table rows and SQL query results as CSS stylesheets defining custom properties that can then be used to customize a static HTML page.

New on this site: a Datasette Tools directory and a search engine that covers documentation, tools, plugins, releases and more. The search engine uses Dogsheep Beta - I wrote about how that works in Building a search engine for datasette.io.

I wrote more about how this site works in Weeknotes: datasette.io, an official project website for Datasette.

Datasette 0.53 - datasette serve --create option, ?column__arraynotcontains= table filter, ?_header=off CSV option and updated links to reflect the new project website.

New Datasette project website - three years after the first release Datasette finally has its own website! The site includes project news, a plugin directory, use-cases and examples. The site is implemented using Datasette custom templates, and the source code is available on GitHub.

Datasette 0.52 - --config is now --setting, new database_actions plugin hook, datasette publish cloudrun --apt-get-install option and several bug fixes.

Datasette 0.51 - A new visual design, plugin hooks for adding navigation options, better handling of binary data, URL building utility methods and better support for running Datasette behind a proxy. Annotated release notes.

Datasette 0.50 - New column actions menu. datasette.client object for plugins to make internal API requests. Improved documentation on deploying Datasette. Annotated release notes.

Datasette 0.49 - JSON API for writable canned queries, path parameters for custom pages. See also Datasette 0.49: The annotated release notes.

Datasette 0.48 - Documentation now lives at docs.datasette.io, improvements to the extra_template_vars, extra_css_urls, extra_js_urls and extra_body_script plugin hooks.

Datasette 0.47 - Datasette can now be installed using Homebrew! brew install simonw/datasette/datasette. Also new: datasette install name-of-plugin and datasette uninstall name-of-plugin commands, and datasette --get '/-/versions.json' to output the result of Datasette HTTP calls on the command-line.

Datasette 0.46 - security fix relating to CSRF protection for writable canned queries, a new logo, new debugging tools, improved file downloads and more.

GraphQL in Datasette with the new datasette-graphql plugin

Two new plugins: datasette-copyable and datasette-insert-api. datasette-copyable adds copy-and-paste export options, and datasette-insert-api lets you create tables and insert or update data by POSTing JSON directly to Datasette.

Datasette 0.45 - Magic parameters for canned queries, a log out feature, improved plugin documentation and four new plugin hooks. See also Datasette 0.45: The annotated release notes.

A cookiecutter template for writing Datasette plugins

Datasette 0.44 - Authentication and permissions, writable canned queries, flash messages, new plugin hooks and much, much more.

Datasette 0.43 - Redesigned register_output_renderer plugin hook and various small improvements and fixes.

Datasette 0.42 - Documented internal methods for plugins to execute read queries against a database.

Datasette 0.41 - New mechanism for creating custom pages, new configuration directory mode, new ?column__notlike= table filter and various other smaller improvements.

Datasette 0.40 - Metadata can now be provided as YAML instead of JSON. Publishing to Zeit Now v1 is no longer supported, but Now v2 support is provided by the new datasette-publish-now plugin. Various bug fixes.

Datasette 0.39 - New base_url configuration option for running Datasette under a different URL prefix, "sort" and "sort_desc" metadata options for setting a default sort order for a table.

Datasette 0.38 - New --memory option for datasete publish cloudrun, Docker image upgraded to SQLite 3.31.1.

Datasette 0.37 - new internal APIs enabling plugins to safely write to databases. Read more here: Datasette Writes.

Datasette 0.36 - new internals documentation for plugins, prepare_connection() now accepts optional database and datasette arguments.

Datasette 0.35 - new .render_template() method for plugins.

Datasette 0.34 - improvements to search, datasette publish cloudrun and datasette package.

Deploying a data API using GitHub Actions and Cloud Run - how to use GitHub Actions and Google Cloud Run to automatically scrape data and deploy the result as an API with Datasette.

Datasette 0.33 - various small improvements.

Building tools to bring data-driven reporting to more newsrooms - some notes on my JSK fellowship so far.

Niche Museums is a new site entirely powered by Datasette, using custom templates and plugins. niche-museums.com, powered by Datasette describes how the site works, and datasette-atom: Define an Atom feed using a custom SQL query describes how the new datasette-atom plugin was used to add an Atom syndication feed to the site.

Datasette 0.32 now uses asynchronous rendering in Jinja templates, which means template functions can perform asynchronous operations such as executing SQL queries. datasette-template-sql is a new plugin uses this capability to add a new custom sql(sql_query) template function.

Datasette 0.31 - the first version of Datasette to support Python 3.8, which means dropping support for Python 3.5.

Datasette 0.30

Single sign-on against GitHub using ASGI middleware talks about the implementation of datasette-auth-github in more detail.

Datasette 0.29 - ASGI, new plugin hooks, facet by date and much, much more...

• datasette-auth-github - a new plugin for Datasette 0.29 that lets you require users to authenticate against GitHub before accessing your Datasette instance. You can whitelist specific users, or you can restrict access to members of specific GitHub organizations or teams.
• datasette-cors - a plugin that lets you configure CORS access from a list of domains (or a set of domain wildcards) so you can make JavaScript calls to a Datasette instance from a specific set of other hosts.

Porting Datasette to ASGI, and Turtles all the way down

The anonymized raw data from the Stack Overflow Developer Survey 2019 has been published in partnership with Glitch, powered by Datasette.

Datasette 0.28 - a salmagundi of new features!

• No longer immutable! Datasette now supports databases that change.
• Faceting improvements including facet-by-JSON-array and the ability to define custom faceting using plugins.
• datasette publish cloudrun lets you publish databases to Google's new Cloud Run hosting service.
• New register_output_renderer plugin hook for adding custom output extensions to Datasette in addition to the default .json and .csv.
• Dozens of other smaller features and tweaks - see the release notes for full details.
• Read more about this release here: Datasette 0.28 - and why master should always be releasable

sqlite-utils: a Python library and CLI tool for building SQLite databases - a partner tool for easily creating SQLite databases for use with Datasette.

Datasette 0.27 - datasette plugins command, newline-delimited JSON export option, new documentation on The Datasette Ecosystem.

Datasette 0.26.1 - SQLite upgrade in Docker image, /-/versions now shows SQLite compile options.

Datasette 0.26 - minor bug fixes, datasette publish now --alias argument.

Fast Autocomplete Search for Your Website - a new tutorial on using Datasette to build a JavaScript autocomplete search engine.

The interesting ideas in Datasette - a write-up of some of the less obvious interesting ideas embedded in the Datasette project.

Datasette 0.25 - New plugin hooks, improved database view support and an easier way to use more recent versions of SQLite.

Datasette 0.24 - a number of small new features

datasette-vega, a new plugin for visualizing data as bar, line or scatter charts

Datasette 0.23.1 - minor bug fixes

Datasette 0.23: CSV, SpatiaLite and more - CSV export, foreign key expansion in JSON and CSV, new config options, improved support for SpatiaLite and a bunch of other improvements

Datasette 0.22.1 bugfix plus we now use versioneer

Datasette 0.22: Datasette Facets

Datasette 0.21: New _shape=, new _size=, search within columns

Exploring the UK Register of Members Interests with SQL and Datasette - a tutorial describing how register-of-members-interests.datasettes.com was built (source code here)

Datasette plugins, and building a clustered map visualization - introducing Datasette's new plugin system and datasette-cluster-map, a plugin for visualizing data on a map

Datasette 0.20: static assets and templates for plugins

Datasette 0.19: plugins preview

Datasette 0.18: units

Datasette 0.15: sort by column

Baltimore Sun Public Salary Records - a data journalism project from the Baltimore Sun powered by Datasette - source code is available here

Cloud-first: Rapid webapp deployment using containers - a tutorial covering deploying Datasette using Microsoft Azure by the Research Software Engineering team at Imperial College London

Analyzing my Twitter followers with Datasette - a tutorial on using Datasette to analyze follower data pulled from the Twitter API

Datasette Publish: a web app for publishing CSV files as an online database

Building a location to time zone API with SpatiaLite, OpenStreetMap and Datasette

Datasette 0.14: customization edition

New in Datasette: filters, foreign keys and search

Datasette: instantly create and publish an API for your SQLite databases