Skip to content
Tutorials Center
My learning

Topics

Lesson 8 of 49 ยท Biological Data and Databases

File formats: FASTA, FASTQ, GFF3, BED and VCF

Read and write the plain-text formats behind genomics: FASTA, FASTQ, GFF3, BED and VCF, with a clear guide to 0-based and 1-based coordinates.

Bioinformatics tools talk to each other through files. A mapper reads FASTQ and writes alignments, a variant caller reads alignments and writes VCF, a genome browser reads GFF3 and BED. These formats were invented by different groups over decades, so each has its own habits. The good news is that all of the common ones are plain text, so you can open them in an editor, read the first lines and know what you have. The bad news is that they disagree about how to count positions, and that disagreement is the source of a large share of real-world bugs.

Four dark code panels labelled FASTA, FASTQ, GFF3 and VCF, with a pill about 0-based and 1-based coordinates
Caption: sequences, reads, features and variants each have a typical format; GFF3 and VCF count from 1 while BED counts from 0.

What you will learn

  • The structure of FASTA, FASTQ, GFF3, BED and VCF records.
  • How 1-based inclusive and 0-based half-open coordinates differ, and how to convert.
  • What compressed and indexed files are for.

Key terms

  • Record: one unit in a file (a sequence, a read, a feature, a variant).
  • Header: lines that describe the file or a record rather than data.
  • 1-based, inclusive: the first base is position 1; an interval includes both ends.
  • 0-based, half-open: the first base is position 0; an interval includes the start and excludes the end.
  • Index: a small companion file that lets a program jump straight to a region of a large file.

FASTA: names and sequences

FASTA is the simplest format. Each record begins with a line starting with >, the header, containing an identifier (everything up to the first space) and an optional description. The following lines hold the sequence, often wrapped at 60 or 80 characters. Proteins use the same layout with amino-acid letters. FASTA has no quality scores and no coordinates; it is the format for references, genes and proteins. A file may contain one record (a genome) or millions (a protein database). Lowercase letters usually indicate soft-masked repeats, and N stands for an unknown base.

FASTQ: reads with qualities

FASTQ stores sequencing reads. Every record is exactly four lines: a header beginning with @, the sequence, a line with just + (optionally repeating the name), and a quality string of the same length as the sequence. The quality characters encode a probability of error for each base. The next lesson decodes them. Paired-end sequencing yields two files, one with the first read of each pair and one with the second, in the same order. Because the quality line may itself start with @, you cannot parse FASTQ by looking for @; you must count lines in fours.

GFF3: features on a sequence

GFF3 (General Feature Format, version 3) describes annotated features such as genes, exons and coding sequences. Lines beginning with # are comments or directives. Every data line has nine tab-separated columns:

  1. seqid: the sequence (chromosome or contig) name
  2. source: the program or database that made the feature
  3. type: a feature type such as gene, mRNA, exon or CDS
  4. start and 5. end: positions, 1-based and inclusive
  5. score: a number or . when absent
  6. strand: +, -, or .
  7. phase: for coding features, 0, 1 or 2 (how many bases to skip to reach a codon boundary), else .
  8. attributes: semicolon-separated key=value pairs, such as ID=gene1;Name=abc

A gene on the minus strand still has start less than or equal to end; the strand column carries the direction. Features are linked into a hierarchy (gene, then mRNA, then exons) using ID and Parent attributes.

BED: simple intervals

BED is the lightweight alternative. It needs at least three columns, chrom, start, end, and may add a name, a score and a strand. Unlike GFF3, BED is 0-based and half-open: the start is the zero-based index of the first base and the end is the index just past the last base. BED is popular for lists of regions: peaks, exons, target intervals. Its logic is the same as Python's slices, so a BED row chr1 100 220 is exactly seq[100:220], which has length end - start = 120.

VCF: variants

The Variant Call Format records differences between samples and a reference. Header lines starting with ## define metadata, one line starting with #CHROM names the columns, and each following line is a variant. The fixed columns are CHROM, POS (1-based), ID, REF (the reference allele), ALT (one or more alternative alleles, comma-separated), QUAL, FILTER and INFO (semicolon-separated annotations). When genotypes are present, a FORMAT column and one column per sample follow, with entries such as 0/1 for a heterozygous call. A single-base substitution has one-letter REF and ALT; an insertion or deletion shows the anchoring base plus the inserted or deleted bases.

Aligned reads and compression

Once reads are placed on a reference they are stored in SAM (a text format with one alignment per line), BAM (its compressed binary form) or CRAM (smaller still). You will meet them in chapter 5. Large text files are usually compressed with gzip, often in the blocked variant bgzip, so that tools can read from the middle. A .tbi, .csi or .fai index lets a tool jump to chr7:55,000,000-55,100,000 without reading the whole file. Always keep the index with its file and rebuild it if you change the data.

The coordinate trap

Both conventions have reasons. The 1-based inclusive style matches how biologists talk ("bases 101 to 220"). The 0-based half-open style makes arithmetic clean, since length is end - start and adjacent intervals meet exactly. The conversion between them is short:

  • GFF3 start..end (1-based inclusive) to BED: bed_start = start - 1, bed_end = end.
  • BED to GFF3: start = bed_start + 1, end = bed_end.
  • Python slice for a GFF3 feature: seq[start - 1:end].

Whenever a tool or a result "looks one base off," suspect a coordinate convention before suspecting biology.

Choosing a format and validating files

A few habits prevent most format problems. First, peek at the first lines (the command head in a terminal, or reading a few lines in Python) before you parse a file. Second, write small validators: check that every FASTQ record has four lines and that the sequence and quality lines have equal length, that every GFF3 line has nine columns, and that positions are integers with start not greater than end. Third, prefer standard tools such as samtools, bcftools and bedtools for conversions and intersections rather than writing your own, because they have been tested on huge files and on edge cases (spliced reads, multi-allelic variants, circular chromosomes). Finally, keep raw files unchanged. Do all processing into new files, so that you can always return to the original data and repeat an analysis.

Common misconceptions

  • "All formats count from 1." BED counts from 0 with an exclusive end; so do Python slices.
  • "FASTQ records start wherever @ appears." Quality characters can be @, so parse in groups of four lines.
  • "A VCF line describes the whole sample." Each line describes one position; samples carry genotypes in later columns.
  • "A file named .fasta is FASTA." Always peek at the first lines.

Try it

Python GFF3 to BED and back to a sequence

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

See the two coordinate systems side by side
Python Read VCF records

Press Run (or Ctrl + Enter) to see the result here.

Program input ยท what you'd type while the program runs

Type the answers before you press Run, one per line. Each time the program reads a line (with input()), it gets the next line from this box.

Check your understanding

Quick check ยท 5 questions

of 5 answered You scored / 5 ยท perfect!

  1. 1. A GFF3 line says the feature spans 101 to 220. What is its BED representation?

    BED is 0-based and half-open: start = 101 - 1 = 100; the end stays 220.
  2. 2. How many lines does one FASTQ record have?

    A FASTQ record is exactly four lines: header, sequence, plus line and qualities.
  3. 3. Which column of a GFF3 line carries the key=value annotations?

    Column 9 holds the attributes such as ID=gene1;Name=abc.
  4. 4. What does the REF column of a VCF record contain?

    REF is the allele in the reference sequence; ALT lists the alternatives.
  5. 5. BED 'chr1 100 220' has what length?

    For half-open intervals the length is end - start = 120.

Summary

  • FASTA stores sequences, FASTQ reads with qualities, GFF3 annotated features, BED simple intervals, VCF variants.
  • GFF3 and VCF use 1-based coordinates; BED and Python slices use 0-based, half-open ones.
  • Convert with bed_start = start - 1, bed_end = end.
  • Compressed, indexed files let tools jump straight to a region.

Next: FASTQ and Phred quality scores