{"ok": true, "database": "tils", "table": "til", "rows": [{"path": "datasette_crawling-datasette-with-datasette.md", "topic": "datasette", "title": "Crawling Datasette with Datasette", "url": "https://github.com/simonw/til/blob/main/datasette/crawling-datasette-with-datasette.md", "body": "I wanted to add the new tutorials on https://datasette.io/tutorials to the search index that is used by the https://datasette.io/-/beta search engine.\n\nTo do this, I needed the content of those tutorials in a SQLite database table. But the tutorials are implemented as static pages in [templates/pages/tutorials](https://github.com/simonw/datasette.io/tree/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/templates/pages/tutorials) - so I needed to crawl that content and insert it into a table.\n\nI ended up using a combination of the `datasette.client` mechanism ([documented here](https://docs.datasette.io/en/stable/internals.html#internals-datasette-client)), [Beautiful Soup](https://www.crummy.com/software/BeautifulSoup/bs4/doc/) and [sqlite-utils](https://sqlite-utils.readthedocs.io/) - all wrapped up in [a Python script](https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/index_tutorials.py) that's now called as part of [the GitHub Actions build process](https://github.com/simonw/datasette.io/blob/9dffe361b0210b9d8b1f2fb820a3f2193f0f2fc7/scripts/build.sh#L35) for the site.\n\nI'm also using [configuration directory mode](https://docs.datasette.io/en/stable/settings.html#config-dir).\n\nHere's the annotated script:\n\n```python\nimport asyncio\nfrom bs4 import BeautifulSoup as Soup\nfrom datasette.app import Datasette\nimport pathlib\nimport sqlite_utils\n\n# This is an async def function because it needs to call await ds.client\nasync def main():\n db = sqlite_utils.Database(\"content.db\")\n # We need to simulate the full https://datasette.io/ site - including all\n # of its custom templates and plugins. On the command-line we would do this\n # by running \"datasette .\" - using configuration directory mode. This is\n # the equivalent of that when constructing the Datasette object directly:\n ds = Datasette(config_dir=pathlib.Path(\".\"))\n # Equivalent of fetching the HTML from https://datasette.io/tutorials\n index_response = await ds.client.get(\"/tutorials\")\n index_soup = Soup(index_response.text, \"html5lib\")\n # We want to crawl the links inside