Skip to content

Historical demonstration · 2021

Using a data catalogue.

Find a dataset by what it describes, then inspect its columns before deciding how to use it.

This short demonstration connects three views: project tables in Snowflake, their metadata in Neo4j, and dataset search in Amundsen. The catalogue makes the datasets and their metadata searchable without starting from the database’s table list.

← Data, evidence & assurance in the Library

Automated project data catalogue · Lawrence Rowland · 2 minutes 49 seconds.

Download the film (MP4)

Follow the data

  1. Start with the stored records. Snowflake holds the project and activity tables in the OXFORD_PORTFOLIOS database.
  2. Inspect the metadata. A Neo4j graph connects the schema, tables and columns: a description of how the data is organised.
  3. Find a useful dataset. Searching for “agency” in Amundsen returns the project and activity datasets. Opening the project dataset exposes its columns, including planned and actual costs.

The useful distinction is between storing data and helping someone find and understand it. A catalogue supplies that second view; judging a dataset’s quality, currency and fitness for a particular decision still takes further work.

Recorded in 2021. Account and contact details are obscured in this viewing copy. The film preserves the interfaces and demonstration as they were; it is not a current setup guide.

Back to Data, evidence & assurance