3. Building a catalogue of datasets

Recipe Overview
Reading Time
60 minutes
Executable Code
Building a catalogue of datasets
FAIRPlus logo
Recipe Type
Maturity Level & Indicator
hover me Tooltip text

3.1. Main Objectives

The main purpose of this recipe is:

To detail the key elements for the creation of a data catalogue to enable data findability in an organisation.

We will cover the following points:

  1. metadata model selection

  2. annotation with controlled vocabularies

  3. ETL

  4. data loading

  5. data indexing

  6. facet oriented searching

  7. minting of stable, persistent and resolvable identifiers

Data Formats












3.2. Graphical Overview

Building and populating a data catalogue

Fig. 3.4 Building and populating a data catalogue

3.3. User Story

For role.Data Scientists, it is essential to be able to action.identify and action.discover datasets of potential relevance in the context of action.data integration and action.meta-analytical work.

For role.Database Managers, a lightweight solution is needed to support a shallow indexing supported fast ingest without intense curation, but good potential for data discovery. Works should rely on approved data standards.

For role.lab scientists, the key is to have a minimal burden when having to action.deposit a dataset to an institutional archive or simply action.register to dataset to the data catalogue.

3.4. Main body of the recipe

3.4.1. What is a Data Catalogue?

A Data Catalogue is a resource meant to allow fast identification of Data set. In keeping with the familiar notion of catalogue, (be it that of an exhibition or that of brand products), the notion of data catalogue needs to be understood as the compendium of short descriptive metadata elements about an actual set of data. The Data Index or Data Catalogue does not store the datasets themselves but provides information about where the datasets can be obtained from. Therefore, Data Catalogues are often used to index the content of ‘Data Repositories and ** Data Archives**, which provide hosting solutions for the actual datasets, which are often organized (but not always)’ around specific data types or data production modalities (e.g. NMR Imaging, Confocal microscopy imaging, Nucleic Acid sequence archives and so on.)

3.4.2. What are the standards supporting establishing a data catalogue?

Data Catalogues have been identified as critical infrastructure and therefore a number of model exist to support their implementation.

  1. DATS: The Data Article Tag Suite model has been developed during the NIH-BD2K projects and underpins the datamed catalogue, the aim of which was to produce a prototype of a Pubmed for Datasets.

  2. DCAT: In the world of semantic web technologies, The W3C DCAT specifications (v1 and the newly released version 2) provide a vocabulary to express data catalogue metadata in RDF.

  3. Schema.org: The vocabulary developed by the consortium of search engines has defined a metadata profile for Dataset, Data Catalogue

3.4.3. How are Data Catalogue populated?

A number data Indexes/Data Catalogue are populated by harvest Dataset metadata from primary Data Repositories or harvesting JSON-LD files served by these same pages for rapid, shallow indexing. The former method is often richer but requires more

3.4.4. What are examples of Data Catalogues?

3.5. Conclusion

This recipe introduced the general concept of data catalogue and why they constitute a key capability to deliver data discoverability.

3.6. References

3.7. Authors