13. File format validation, FASTQ example

Recipe Overview
Reading Time
15 minutes
Executable Code
Validating file format - FASTQ example
FAIRPlus logo
Recipe Type
Maturity Level & Indicator
not applicable
hover me Tooltip text

13.1. Main Objectives

The main purpose of this recipe is to:

  • provide a FASTQ file validation solution

  • propose a general file validation workflow.

13.2. Graphical Overview

13.3. User Stories

The table below lists common file validation use cases. This recipe provides solutions with FASTQ files 1 as an example.

As a ..

I want to ..

So that I can ..

Data owner

Validate my sequencing files before depositing to public archives

Reduce the risk of submitting invalid files or submission rejection

Data consumer

Validate files before running analysis

Avoid wasting time and resource processing corrupted files

Data consumer

Integrate file format validation into my data process pipeline

Build a more reproducible and error-proof pipeline

Data librarian

Check files downloaded from unknown sources before deposition

Ensure the file is usable in the future.

13.4. FAIRification Objectives, Inputs and Outputs




Format validation

FASTQ file

Validation results

13.5. Table of Data Standards

Data Formats




Compressed Format

FASTQ is the de facto sequencing file format and one of the most common file formats in bioinformatics analysis 2, 4. Researchers receive FASTQ files from various sources. These files are used intensively in automated bioinformatics analysis pipelines. Therefore, it is important to validate FASTQ files to improve the data reusability and build error-proof data analysis processes.

FASTQ validators detect truncated reads, base calls and quality score mismatches, invalid encoding, etc. For paired-end reads, they also check if the forward reads match with the reverse reads. Most validators can process different FASTQ variants automatically and handle compressed FASTQ files.

FASTQ-utils is an open-source software to validate and process FASTQ files. It has been applied in the European Nucleotide Archive(ENA), and several research initiatives.

This recipe provides an example of validating FASTQ files with FASTQ-utils on MacOS and Linux machines.


⚠️ Quality control is out of the scope of file format validation.

13.5.1. Requirements

The users are expected to be comfortable with Unix-based OS and basic Bash programming syntax and commands.





Package manager for installing validators



FASTQ validator



File downloader


13.5.2. Step 1: Install fastq-utils

The command below installs fastq-utils via Conda. It is also possible to install fastq-utils from the source code 3.

conda install -c bioconda fastq_utils

13.5.3. Step 2: Get example file for testing*


  • Users can skip this step and test with their own files._

In this step, we download example FASTQ files from ENA for testing. The first example file is a single read file, the other ones are paired-end read files.

Example 1: Get single read FASTQ file

The command below downloads an Ion Torrent S5 fastq file from ENA. This file is the whole genome sequencing file of SARS-CoV-2. The complete file is 192Mb.

wget -c ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR121/077/SRR12132977/SRR12132977.fastq.gz

Users can inspect the fastq.gz file using gzip -cd SRR12132977.fastq.gz | head -8. Below is the header of this FASTQ file.

@SRR12132977.1 1/1
@SRR12132977.2 2/1

Example 2: Get paired-read FASTQ files

The command below downloads Illumina iSeq 100 paired end sequencing files from ENA. These files are raw sequence reads of a SARS-CoV-2 sample. Each file is 26 Mb.

wget -c \
ftp://ftp.sra.ebi.ac.uk/vol1/fastq/SRR115/044/SRR11542244/SRR11542244_1.fastq.gz \

Below is the headers of the two files. The read pairs info is listed in the read IDs.

# Header of the forward read, SRR11542244_1.fastq.gz
@SRR11542244.1 1/1
@SRR11542244.2 2/1

# Header of the reverse read, SRR11542244_2.fastq.gz
@SRR11542244.1 1/2
@SRR11542244.2 2/2

13.5.4. Step 3: Perform validation

The command below validates the single read file in Example 1.

fastq_info -r SRR12132977.fastq.gz

Below are the validation results. fastq-utils returns the number of reads, read length details, and encoding info. Field Quality encoding indicates the fastq file variant. FASTQ-utils returns OK for a valid fastq file. Otherwise, it will return the validation details in the Error message.

Skipping check for duplicated read names
Number of reads: 1919741
Quality encoding range: 34 77
Quality encoding: 33
Read length: 25 352 215

The validation of paired end reads is similar to single read file validation.

fastq_info SRR11542244_1.fastq.gz SRR11542244_2.fastq.gz

Here are the validation results.

fastq_utils 0.23.0
Scanning and indexing all reads from SRR11542244_1.fastq.gz
700000Scanning complete.

Reads processed: 733611
Memory used in indexing: ~47 MB
File SRR11542244_1.fastq.gz processed
Next file SRR11542244_2.fastq.gz
Number of reads: 733611
Quality encoding range: 35 70
Quality encoding: 33
Read length: 35 101 96

fastq_util also provides additional arguments to tune the validation:

-s: to validate if reads in two files have the same ordering.

-r: to skip duplicated read names validation. It uses less memory and runs faster.

-e: to allow empty files pass the validation

-q: not to fail if the encoding can’t be decided. Error messages for invalid files

FASTQ-utils returns an error message with the location of invalid lines and type of errors if the files are invalid. Below are examples of error messages.

  • Invalid file example 1, duplicated reads

    ERROR: Error in file SRR11542244_2.fastq: line 16: duplicated sequence SRR11542244.5 5/

  • Invalid file example 2, wrong base call encoding

    ERROR: Error in file SRR11542244_2.fastq: line 5: invalid character ‘e’ (hex. code:’65’), expected ACGTacgt0123nN.

13.5.5. FASTA-utils feature summary

The table lists technical considerations when selecting the validator, including basic validation function, performance, interface, etc. It also provides a detailed summary of fastq-utils features.


Validation content



Basic validation

4-line format

Check if the FASTQ file is a 4-line file


Character encoding

Check if the base calls and quality score encoding are correct.


Read length

Check if the length of the base calls are the same as that of the quality scores


File truncation

Check if the file is truncated or not


Paired-end reads validation

Deinterleaved paired reads

Validate when the forward and reverse reads are in two files.


Interleaved “8-line” files

Validate when the forward and reverse reads are listed together as an 8-line file


Compressed file validation


Validate compressed fastq files, with extension fastq.gz


FASTQ variants* validation


Validate the fastq-illumina format



Validate the fastq-sanger format



Validate the fastq-solexa format







Archieve compatiablity


File validated can be submitted to the ENA archive.



File validated can be submitted to Array Express.



File validated can be submitted to the SRA archive.



Command line interface

Can be used in shell and intergerated in pipe commands





Commercial use

Can be used for commercial purpose



Open source

Source code available on public platforms


*See details in the [FASTQ specification recipe]( TODO include link).

13.6. Conclusion

In this recipe, we have shown how to validate fastq files, and proposed indicators to evaluate a FASTQ validator. We also identified common file validation related use cases and provided a general file validation workflow. This recipe can be expanded to other file formats and other use cases.

13.7. References

13.8. Authors