RSS feeds for Dave's wiki

SAMTools

Fri, 06 Apr 2012 20:31:33 -0500

-
Converting a SAM file to a BAM file

-
Sorting a BAM file

-
Creating a BAM index file

-
Converting a BAM file to a SAM file

-
Extracting SAM entries mapping to a specific loci

-
Simple stats using samtools flagstat

-
Interpreting the BAM flags

-
Extracting only the first read from paired end BAM files

-
Filtering out unmapped reads in BAM files

-
Creating FASTQ files from a BAM file

-
More information

-
Basic usage

samtools

15" >samtools <command> [options]

samtools

samtools <command> [options]

samtools

6" >Program: samtools (Tools for alignments in the SAM format)

samtools

Program: samtools (Tools for alignments in the SAM format)

shell

17" >head test.sam

shell

2" >head test.sam

samtools

18" >samtools view -bT reference.fa test.sam > test.bam

samtools

3" >samtools view -bT reference.fa test.sam > test.bam

samtools

19" >samtools view -bS test.sam > test.bam

samtools

4" >samtools view -bS test.sam > test.bam

samtools

20" >samtools sort test.bam test_sorted

samtools

5" >samtools sort test.bam test_sorted

samtools

21" >samtools index test_sorted.bam test_sorted.bai

samtools

6" >samtools index test_sorted.bam test_sorted.bai

samtools

22" >samtools view -h NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.bam > NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.sam

samtools

7" >samtools view -h NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.bam > NA06984.chrom16.ILLUMINA.bwa.CEU.low_coverage.20100517.sam

samtools

23" >samtools view -L test.bed test.bam | awk '$2 != 4 {print}'

samtools

8" >samtools view -L test.bed test.bam | awk '$2 != 4 {print}'

samtools flagstat

24" >samtools flagstat NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam

samtools flagstat

9" >samtools flagstat NA06984.chrom20.ILLUMINA.bwa.CEU.low_coverage.20111114.bam

- Below is a very raw Perl script that may help interpret BAM flags. I make the assumption that when a read or its mate is not flagged as unmapped (i.e. having a 0 value), there is information regarding its strandedness.

+ Below is a very raw Perl script that may help interpret BAM flags. I make the assumption that when a read or its mate is not flagged as unmapped (i.e. having a 0 value), there is information regarding its strandedness. Also if a read is flagged as not having a proper pair, I ignore the flags that store information for the mate.

-
bam_flag.pl
25" >#!/usr/bin/perl

+
bam_flag.pl
10" >#!/usr/bin/perl

+ my $proper_pair = '0';

+

+ print "$binary\n";

+ if ($i == 1){

+ if ($flag == 1){

+ $proper_pair = '1';

+ }

+ next if $proper_pair == 0;

samtools

26" >samtools view -h -f 0x0040 test.bam > test_first_pair.sam

samtools

11" >samtools view -h -f 0x0040 test.bam > test_first_pair.sam

samtools

7" >samtools view -h -F 4 blah.bam > blah_only_mapped.sam

samtools

12" >samtools view -h -F 4 blah.bam > blah_only_mapped.sam

samtools

28" >bam2fastq -o blah_unaligned.fastq --no-aligned blah.bam

samtools

13" >bam2fastq -o blah_unaligned.fastq --no-aligned blah.bam

samtools

29" >bam2fastq -o blah_aligned.fastq --no-unaligned blah.bam

samtools

14" >bam2fastq -o blah_aligned.fastq --no-unaligned blah.bam

Personal

Fri, 06 Apr 2012 00:27:50 -0500

+

+ Personality tests and others: http://similarminds.com/index.html

RACE

Wed, 04 Apr 2012 08:02:32 -0500

+ Rapid Amplification of cDNA Ends (RACE) is a technique used to obtain the full length sequence of an RNA transcript.

+

+ RACE results in the production of a cDNA copy of the RNA sequence of interest, produced through reverse transcription, followed by PCR amplification of the cDNA copies.

+

+ The amplified cDNA copies are then sequenced and RACE can provide the sequence of an RNA transcript from a small known sequence within the transcript to the 5' end (5' RACE-PCR) or 3' end (3' RACE-PCR) of the RNA.

+

+ The first step in RACE is to use reverse transcription to produce a cDNA copy of a region of the RNA transcript. In this process, an unknown end portion of a transcript is copied using a known sequence from the center of the transcript. The copied region is bounded by the known sequence, and either the 5' or 3' end.

+

+ The protocols for 5' or 3' RACES differ slightly. 5' RACE-PCR begins using mRNA as a template for a first round of cDNA synthesis (or reverse transcription) reaction using an anti-sense (reverse) oligonucleotide primer that recognizes a known sequence in the gene of interest; the primer is called a gene specific primer (GSP), and it copies the mRNA template in the 3' to the 5' direction to generate a specific single-stranded cDNA product. Following cDNA synthesis, the enzyme terminal deoxynucleotidyl transferase (TdT) is used to add a string of identical nucleotides, known as a homopolymeric tail, to the 3' end of the cDNA. (There are some other ways to add the 3'-terminal sequence for the first strand of the de novo cDNA synthesis which are much more efficient than homopolymeric tailing, but the sense of the method remains the same). A PCR reaction is then carried out, which uses a second anti-sense gene specific primer (GSP2) that binds to the known sequence, and a sense (forward) universal primer (UP) that binds the homopolymeric tail added to the 3' ends of the cDNAs to amplify a cDNA product from the 5' end.

+

+ 3' RACE-PCR uses the natural polyA tail that exists at the 3' end of all eukaryotic mRNAs for priming during reverse transcription, so this method does not require the addition of nucleotides by TdT. cDNAs are generated using an Oligo-dT-adaptor primer (a primer with a short sequence of deoxy-thymine nucleotides)that complements the polyA stretch and adds a special adaptor sequence to the 5' end of each cDNA. PCR is then used to amplify 3' cDNA from a known region using a sense GSP, and an anti-sense primer complementary to the adaptor sequence.

Bioinformatics

Wed, 04 Apr 2012 07:44:34 -0500

+
RACE

PCR

Wed, 04 Apr 2012 07:43:19 -0500

+

+
qPCR

+

+ The concept of qPCR is simple: amplification products are measured as they are produced using a fluorescent label. During amplification, a fluorescent dye binds, either directly or indirectly via a labeled hybridizing probe, to the accumulating DNA molecules, and fluorescence values are recorded during each cycle of the amplification process. The fluorescence signal is directly proportional to DNA concentration over a broad range, and the linear correlation between PCR product and fluorescence intensity is used to calculate the amount of template present at the beginning of the reaction. The point at which fluorescence is first detected as statistically significant above the baseline or background, is called the threshold cycle or Ct Value.

Unix

Sat, 24 Mar 2012 10:42:38 -0500

-
PS1 notation

-
Finding a zombie process

-
Termination signals

-
SIGTERM

-
SIGINT

-
SIGQUIT

-
SIGKILL

-
SIGHUP

-
screen

-
svn tutorial

-
Shell scripting

-
if-elif-else statement

-
For loop

-
qstat

-
Ubuntu

-
dos2unix

-
Ubuntu - installing g++, make, openssh server, zlib to compile stuff on Ubuntu

-
Grub2

-
Renaming several files in bash

-
Automating ftp

-
gnuplot

-
tr

-
.bashrc

-
awk

-
cron

-
Computing clusters

-
bottom_of_the_page

-
Setting up SSH public and private keys

ps1

7" >\a     an ASCII bell character (07)

ps1

\a     an ASCII bell character (07)

-
if_elif_else.sh
8" >if [ -f exist.txt ]; then

+
if_elif_else.sh
1" >if [ -f exist.txt ]; then

bash_and.sh

9" >if [ -f total_snp -a -s total_snp ]; then

bash_and.sh

2" >if [ -f total_snp -a -s total_snp ]; then

for

40" >#!/bin/bash

for

33" >#!/bin/bash

-
basename.sh
1" >#!/bin/bash -f

+
basename.sh
34" >#!/bin/bash -f

-
ftp.sh
42" >#!/bin/sh

+
ftp.sh
35" >#!/bin/sh

.bashrc

43" ># User specific aliases and functions

.bashrc

6" ># User specific aliases and functions

+

+ if -d "$HOME/bin" ; then

+ PATH="$PATH:$HOME/bin"

+ fi

Biology

Fri, 23 Mar 2012 09:15:40 -0500

-
Genetic Code

-
Mendelian Genetics - Patterns of Inheritance and Single-Gene Disorders

-
Gene Models

-
RNA

-
Hormones

-
Terminology

-
Glossary

-
Transcriptomics

-
Promoter

-
Transcription factor

-
DNA-binding domain

-
Technologies

-
PCR and RT

-
Gene cloning using plasmids

-
Tiling array

-
ChIP-chip

-
Mass spec

-
Epigenetics

-
Chromatin remodelling

-
DNA methylation

-
Polycomb-group proteins

-
Small RNAs

-
Dicer and drosha

-
siRNA

-
RNase H

-
Awesome animations for learning DNA repair

-
General biology

- In recent years, researchers have discovered a great deal about chromatin remodeling, including the roles that different protein complexes, histone variants, and biochemical modifications play in this process. However, a great deal remains to be learned before chromatin remodeling is fully understood.

+ In recent years, researchers have discovered a great deal about chromatin remodeling, including the roles that different protein complexes, histone variants, and biochemical modifications play in this process.

Various molecules called chromatin remodelers provide the mechanism for modifying chromatin and allowing transcription signals to reach their destinations on the DNA strand. Currently, investigators know that chromatin remodelers are large, multiprotein complexes that use the energy of ATP hydrolysis to mobilize and restructure nucleosomes. Recall that nucleosomes wrap 146 base pairs of DNA in approximately 1.7 turns around a histone-octamer disk, and the DNA inside each nucleosome is generally inaccessible to DNA-binding factors. Remodelers are thus necessary to provide access to the underlying DNA to enable transcription, chromatin assembly, DNA repair, and other processes. Just how remodelers convert the energy of ATP hydrolysis into mechanical force to mobilize the nucleosome, and how different remodeler complexes select which nucleosomes to move and restructure, remains unknown, however.

Remodelers are partitioned into five families, each with specialized biological roles. Nonetheless, all remodelers contain a subunit with a conserved ATPase domain. In addition to the conserved ATPase, each remodeler complex also possesses unique proteins that specialize it for its unique biological role. However, because all remodelers move nucleosomes and all such movement is ATP dependent, mobilization is most likely a property of the conserved ATPase subunit.

The ATPase domains of remodelers are similar in sequence and structure to known DNA-translocating proteins in viruses and bacteria. Recent evidence from the SWI/SNF and ISWI remodeler families has also revealed that remodeler ATPases are directional DNA translocases that are capable of the directional pumping of DNA. But how is this property applied to nucleosomes? It seems that the ATPase binds approximately 40 base pairs inside the nucleosome, from which location it pumps DNA around the histone-octamer surface. This enables the movement of the nucleosome along the DNA, thus permitting the exposure of the DNA to regulatory factors.

The additional domains and proteins that are attached to the ATPase are important for nucleosome selection, and they also help regulate ATPase activity. These attendant proteins bind to histones and nucleosomal DNA, and their binding to these molecules is affected by the histone modification state. The modification state helps determine whether the nucleosome is an appropriate substrate for a remodeler complex.

The composition of nucleosomes is not set in stone, however. Indeed, canonical histones can themselves be replaced by histone variants or modified by specific enzymes, thereby making the surrounding DNA more or less accessible to the transcriptional machinery.

So far, a number of histone variants have been found and localized to specific areas of chromatin. For instance, H2A.Z is a variant of H2A and is often enriched near relatively inactive gene promoters. Interestingly, H2A.Z does not take its place during replication when the chromatin structure is established. Instead, the chromatin remodeling complex SWR1 catalyzes an ATP-dependent exchange of H2A in the nucleosome for H2A.Z.

CENP-A is another known histone variant that has been found to be associated with centromeres. Originally localised to the centromere through immunofluorescence studies, CENP-A was believed to be involved in centromeric activity during cell division. But, once the CENP-A protein was isolated and sequenced, it was shown to have sequence homology to H3, suggesting that CENP-A actually replaces canonical H3 near the centromere. Some experiments suggest that these variant histones that occur in particular areas of the genome may assist in the specific regulation of chromatin behavior and gene transcription from these areas.

Specifically, histone modification involves covalent bonding of various functional groups to the free nitrogens in the R-groups of lysines in the N-terminal tail. Early research has linked differing levels of acetylation and methylation on the histones to altered rates of DNA transcription. While the most common additions are acetylation and methylation of lysine residues, many more types of modifications have also been observed, including phosphorylation, a common posttranslational modification. The different types of modifications, which have been called the "histone code," are put in place by a variety of different enzymes, many of which have yet to be fully characterized. Thus, the story of the remodeling machinery continues to be told through a variety of experiments, and much remains to be revealed.

Because eukaryotic DNA is tightly wrapped around nucleosomes and the positive charges of the histones tightly bind the negative charges of the DNA, nucleosomes essentially act as a physical barrier to transcription factors that need to bind to certain regions of DNA. However, specific acetylations can remove the positive charge on the lysine amino group that is acetylated, so the nucleosome "loses its grip" on the DNA. This modification results in a loosening of the coil.

Other remodeling enzyme complexes actually slide the nucleosomes along the DNA to clear them from the promoter regions. In this case, the remodeling enzymes use the energy from ATP to regulate nucleosome movement. For example, prior to transcription in yeast, one of the major types of chromatin remodeling machines, called the SWI/SNF and SAGA histone acetylase complex, is recruited to the yeast HO gene promoter by the SWI5 activator. Activator-dependent chromatin modification then moves the nucleosome out of the way so that RNA polymerase II can reach the promoter regions of the DNA.

Chromatin remodeling activity by SWI/SNF or other remodeling machines can also be required for recruiting additional chromatin remodeling activity to the site, as well as additional downstream sites. Modifications at a promoter can occur in multiple steps that are independently regulated, and additional modifications can occur stepwise stretching from the point of the first modification along the DNA strand in a downstream direction toward the promoter. These modifications open up an elongated region of active chromatin and allow for a wide range of intermediate, transcriptionally inactive states for the eukaryotic promoter. Promoters can also be poised with RNA polymerase bound but not elongating the mRNA; in yeast, up to 15% of sites have such stalled transcription. Changes in gene expression during the specific developmental stages of an organism or cell coincide with fluctuations in the levels of each of the specific protein complexes involved in chromatin remodeling.

Varying levels and types of histone modifications have been shown to correlate with levels of chromatin activation. For example, one group of researchers used antibody-based immunoprecipitation studies to determine that acetylation of histone H3 and methylation at lysine residue K4 appeared to coincide with each other. They also coincided during transcriptional activation in chicken embryos, while methylation at lysine residue K9 marked inactive chromatin.

Another means by which transcription is controlled is through methylation of the DNA strand itself. Not to be confused with histone methylation, methylation of the DNA strand involves cytosine bases of eukaryotic DNA being converted to 5-methylcytosine, resulting in the repression of transcription, particularly in vertebrates and plants. The altered cytosine residues are usually immediately adjacent to a guanine nucleotide, resulting in two methylated cytosine residues set diagonally to one another on opposing DNA strands.

Heavily methylated regions of DNA with elevated concentrations of these so-called CpG groups are often found near transcription start sites. In an interestingly coordinated process, proteins that bind to methylated DNA also form complexes with proteins involved in deacetylation of histones. Therefore, when the DNA is in a methylated state, nearby histones are deacetylated, resulting in compact, semipermanently silent chromatin. Likewise, demethylated DNA does not draw deacetylating enzymes to the histones, but it often attracts histone acetyltransferases, allowing histones to remain acetylated and promoting transcription.

Storage of eukaryotic DNA in small, compact nuclei requires that this DNA be tightly coiled and compacted in the form of chromatin. However, the structure of chromatin also appears to serve a second, possibly more important role, in that it gives eukaryotic cells the capability to exert complex levels of control over gene expression.

Chromatin and the DNA sequences it contains are constantly undergoing modifications, thereby periodically exposing different regions of DNA to transcription factors and RNA polymerases. The cumulative effects of these changes are various states of transcriptional control and the ability of eukaryotic cells to turn genes on and off as needed. This complexity provides eukaryotes with a means of making the most of a relatively small number of genes. However, much research remains to be performed before investigators precisely understand how the many mechanisms of chromatin remodeling operate, as well as how they work together to result in the complex patterns of gene expression characteristic of eukaryotic cells.

UCSC genome browser

Thu, 15 Mar 2012 07:33:50 -0500

+

+
Notes for track annotation

+

+ Extracted from http://genome-source.cse.ucsc.edu/gitweb/?p=kent.git;a=blob_plain;f=src/hg/makeDb/trackDb/README;hb=HEAD

+

+ Currently there are seventeen different track types:

+ axt, bed, chain, clonePos, ctgPos, expRatio, genePred,

+ maf, netAlign, psl, rmsk, sample, wigMaf, wig, bedGraph,

+ chromGraph, bigBed

+

+ 18. type bigWig lower upper

+ This uses a binary indexed file rather than a database table. It is, other

+ than substituting "bigWig" for "wig" in the type line, the same as the

+ "type wig" explained above. The database loading procedure is

+ hgBbiDbLink hg19 myLocalBigWig /gbdb/hg19/bbi/myLocalBigWig.bw

+ The file name given should be a full path name that can be accessed

+ by the Apache WEB server. Or, the file name can also be a

+ valid URL to the bw file.

+ This loader is equivalent to:

+ hgsql hg19 -e 'drop table if exists myLocalBigWig; \

+ create table myLocalBigWig (fileName varchar(255) not null); \

+ insert into myLocalBigWig values

+ ("/gbdb/hg19/bbi/myLocalBigWig.bw");'

+

+ CONTAINER TRACKS

+ ====

+ Container tracks are yet another way of grouping tracks together. The top

+ level container track includes a tag of the form

+ container

+ where currently the only container-type is "multiWig" which may only contain

+ wig and bigWig tracks. The advantage of using "container multiWig" instead

+ of a composite track is that container multiWig supports the "aggregate" tag

+ that allows multiple wiggle graphs to be drawn in the same vertical space.

+

+ Here's an example of a container:

+

+ track wgEncodeRegMarkPromoter

+ container multiWig

+ shortLabel Promoter H3K4Me3

+ longLabel ENCODE Promoter Histone Mark (H3K4Me3) on 9 Cell Lines

+ type wig 0 30000

+ viewLimits 0:100

+ visibility full

+ maxHeightPixels 100:30:11

+ aggregate transparentOverlay

+ showSubtrackColorOnUi on

+ windowingFunction mean

+ priority 1.4

+

+ track wgEncodeRegMarkPromoterGm12878

+ shortLabel Gm12878

+ longLabel Promoter Histone Mark (H3K4Me3) on Gm12878 cells from ENCODE

+ parent wgEncodeRegMarkPromoter

+ type wig 0.00 100593.50

+ color 255,128,128

+

+ track wgEncodeRegMarkPromoterH1hesc

+ shortLabel H1 ES

+ longLabel Promoter Histone Mark (H3K4Me3) on H1 cells from ENCODE

+ parent wgEncodeRegMarkPromoter

+ type wig 0.00 12434.00

+ color 255,212,128

+

+ track wgEncodeRegMarkEnhH3k4me3Hepg2

+ shortLabel HepG2

+ longLabel Promoter Histone Mark (H3K4Me3) on HepG2 cells from ENCODE

+ parent wgEncodeRegMarkPromoter

+ type wig 0.00 4927.00

+ color 212,255,128

+

+ Note that subtracks of a container use the "parent" tag to indicate the container they belong to,

+ just as composite tracks do. Other tags associated with a container are:

+ o aggregate - Allows the subtracks to be drawn on top of each other in the same vertical

+ space. Possible values are: none, transparentOverlay, solidOverlay

+ o showSubtrackColorOnUi on - Puts a small patch of color by the list of subtracks in

+ the track UI page.

+

HOMER

Thu, 15 Mar 2012 01:10:40 -0500

+ Software for motif discovery and ChIP-Seq analysis

+

+ http://biowhat.ucsd.edu/homer/introduction/basics.html

IGV

Wed, 14 Mar 2012 00:41:54 -0500

- 2 class="showhide_heading" id="Reference_commands">Reference commands2>

+ 3 class="showhide_heading" id="Reference_commands">Reference commands3>

+

+
Visualising BAM files using IGV

+