1 Introduction

The missMethyl package contains functions to analyse methylation data from Illumina’s HumanMethylation450 and MethylationEPIC beadchip. These arrays are a cost-effective alternative to whole genome bisulphite sequencing, and as such are widely used to profile DNA methylation. Specifically, missMethyl contains functions to perform SWAN normalisation (Maksimovic, Gordon, and Oshlack 2012), perform differential methylation analysis using RUVm (Maksimovic et al. 2015), differential variability analysis (Phipson and Oshlack 2014) and gene set analysis (Phipson, Maksimovic, and Oshlack 2016). As our lab’s research into specialised analyses of these arrays continues we anticipate that the package will be continuously updated with new functions.

Raw data files are in IDAT format, which can be read into R using the minfi package (Aryee et al. 2014). Statistical analyses are usually performed on M-values, and \(\beta\) values are used for visualisation, both of which can be extracted from objects, which is a class of object created by minfi. For detecting differentially variable CpGs we recommend that the analysis is performed on M-values. All analyses described here are performed at the CpG site level.

2 Reading data into R

We will use the data in the minfiData package to demonstrate the functions in missMethyl. The example dataset has 6 samples across two slides. The sample information is in the targets file. An essential column in the targets file is the Basename column which tells where the idat files to be read in are located. The R commands to read in the data are taken from the minfi User’s Guide. For additional details on how to read the IDAT files into R, as well as information regarding quality control please refer to the minfi User’s Guide.

library(missMethyl)
library(limma)
library(minfi)

library(minfiData)
baseDir <- system.file("extdata", package = "minfiData")
targets <- read.metharray.sheet(baseDir)

## [1] "/home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/SampleSheet.csv"

targets[,1:9]

##   Sample_Name Sample_Well Sample_Plate Sample_Group Pool_ID person age sex
## 1    GroupA_3          H5         <NA>       GroupA    <NA>    id3  83   M
## 2    GroupA_2          D5         <NA>       GroupA    <NA>    id2  58   F
## 3    GroupB_3          C6         <NA>       GroupB    <NA>    id3  83   M
## 4    GroupB_1          F7         <NA>       GroupB    <NA>    id1  75   F
## 5    GroupA_1          G7         <NA>       GroupA    <NA>    id1  75   F
## 6    GroupB_2          H7         <NA>       GroupB    <NA>    id2  58   F
##   status
## 1 normal
## 2 normal
## 3 cancer
## 4 cancer
## 5 normal
## 6 cancer

targets[,10:12]

##    Array      Slide
## 1 R02C02 5723646052
## 2 R04C01 5723646052
## 3 R05C02 5723646052
## 4 R04C02 5723646053
## 5 R05C02 5723646053
## 6 R06C02 5723646053
##                                                                                Basename
## 1 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646052/5723646052_R02C02
## 2 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646052/5723646052_R04C01
## 3 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646052/5723646052_R05C02
## 4 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646053/5723646053_R04C02
## 5 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646053/5723646053_R05C02
## 6 /home/biocbuild/bbs-3.8-bioc/R/library/minfiData/extdata/5723646053/5723646053_R06C02

rgSet <- read.metharray.exp(targets = targets)

The data is now an RGChannelSet object and needs to be normalised and converted to a MethylSet object.

3 Subset-quantile within array normalization (SWAN)

SWAN (subset-quantile within array normalization) is a within-array normalization method for Illumina 450k & EPIC BeadChips. Technical differencs have been demonstrated to exist between the Infinium I and Infinium II assays on a single Illumina HumanMethylation array (Bibikova et al. 2011, Dedeurwaerder, Defrance, and Calonne (2011)). Using the SWAN method substantially reduces the technical variability between the assay designs whilst maintaining important biological differences. The SWAN method makes the assumption that the number of CpGs within the 50bp probe sequence reflects the underlying biology of the region being interrogated. Hence, the overall distribution of intensities of probes with the same number of CpGs in the probe body should be the same regardless of assay type. The method then uses a subset quantile normalization approach to adjust the intensities of each array (Maksimovic, Gordon, and Oshlack 2012).

SWAN can take a MethylSet, RGChannelSet or MethyLumiSet as input. It should be noted that, in order to create the normalization subset, SWAN randomly selects Infinium I and II probes that have one, two and three underlying CpGs; as such, we recommend using set.seed before to ensure that the normalized intensities will be identical, if the normalization is repeated.

The technical differences between Infinium I and II assay designs can result in aberrant beta value distributions (Figure (???)(fig:betasByType), panel “Raw”). Using SWAN corrects for the technical differences between the Infinium I and II assay designs and produces a smoother overall \(\beta\) value distribution (Figure (???)(fig:betasByType), panel “SWAN”).

mSet <- preprocessRaw(rgSet)

mSetSw <- SWAN(mSet,verbose=TRUE)

## [SWAN] Preparing normalization subset
## 450k
## [SWAN] Normalizing methylated channel
## [SWAN] Normalizing array 1 of 6
## [SWAN] Normalizing array 2 of 6
## [SWAN] Normalizing array 3 of 6
## [SWAN] Normalizing array 4 of 6
## [SWAN] Normalizing array 5 of 6
## [SWAN] Normalizing array 6 of 6
## [SWAN] Normalizing unmethylated channel
## [SWAN] Normalizing array 1 of 6
## [SWAN] Normalizing array 2 of 6
## [SWAN] Normalizing array 3 of 6
## [SWAN] Normalizing array 4 of 6
## [SWAN] Normalizing array 5 of 6
## [SWAN] Normalizing array 6 of 6

par(mfrow=c(1,2), cex=1.25)
densityByProbeType(mSet[,1], main = "Raw")
densityByProbeType(mSetSw[,1], main = "SWAN")

Figure 1: Beta value dustributions
Density distributions of beta values before and after using SWAN.

4 Filter out poor quality probes

Poor quality probes can be filtered out based on the detection p-value. For this example, to retain a CpG for further analysis, we require that the detection p-value is less than 0.01 in all samples.

detP <- detectionP(rgSet)
keep <- rowSums(detP < 0.01) == ncol(rgSet)
mSetSw <- mSetSw[keep,]

5 Extracting Beta and M-values

Now that the data has been SWAN normalised we can extract \(\beta\) and M-values from the object. We prefer to add an offset to the methylated and unmethylated intensities when calculating M-values, hence we extract the methylated and unmethylated channels separately and perform our own calculation. For all subsequent analysis we use a random selection of 20000 CpGs to reduce computation time.

mset_reduced <- mSetSw[sample(1:nrow(mSetSw), 20000),]
meth <- getMeth(mset_reduced)
unmeth <- getUnmeth(mset_reduced)
Mval <- log2((meth + 100)/(unmeth + 100))
beta <- getBeta(mset_reduced)
dim(Mval)

## [1] 20000     6

par(mfrow=c(1,1))
plotMDS(Mval, labels=targets$Sample_Name, col=as.integer(factor(targets$status)))
legend("topleft",legend=c("Cancer","Normal"),pch=16,cex=1.2,col=1:2)

Figure 2: MDS plot
A multi-dimensional scaling (MDS) plot of cancer and normal samples.

An MDS plot (Figure 2) is a good sanity check to make sure samples cluster together according to the main factor of interest, in this case, cancer and normal.

6 Testing for differential methylation using

To test for differential methylation we use the limma package (Smyth 2005), which employs an empirical Bayes framework based on Guassian model theory. First we need to set up the design matrix. There are a number of ways to do this, the most straightforward is directly from the targets file. There are a number of variables, with the status column indicating cancer/normal samples. From the person column of the targets file, we see that the cancer/normal samples are matched, with 3 individuals each contributing both a cancer and normal sample. Since the limma model framework can handle any experimental design which can be summarised by a design matrix, we can take into account the paired nature of the data in the analysis. For more complicated experimental designs, please refer to the limma User’s Guide.

group <- factor(targets$status,levels=c("normal","cancer"))
id <- factor(targets$person)
design <- model.matrix(~id + group)
design

##   (Intercept) idid2 idid3 groupcancer
## 1           1     0     1           0
## 2           1     1     0           0
## 3           1     0     1           1
## 4           1     0     0           1
## 5           1     0     0           0
## 6           1     1     0           1
## attr(,"assign")
## [1] 0 1 1 2
## attr(,"contrasts")
## attr(,"contrasts")$id
## [1] "contr.treatment"
## 
## attr(,"contrasts")$group
## [1] "contr.treatment"

Now we can test for differential methylation using the lmFit and eBayes functions from limma. As input data we use the matrix of M-values.

fit.reduced <- lmFit(Mval,design)
fit.reduced <- eBayes(fit.reduced)

The numbers of hyper-methylated (1) and hypo-methylated (-1) can be displayed using the decideTests function in limma and the top 10 differentially methylated CpGs for cancer versus normal extracted using topTable.

summary(decideTests(fit.reduced))

##        (Intercept) idid2 idid3 groupcancer
## Down          7052     0   125         653
## NotSig        3265 20000 19865       18855
## Up            9683     0    10         492

top<-topTable(fit.reduced,coef=4)
top

##               logFC    AveExpr        t      P.Value  adj.P.Val        B
## cg19677522 3.992371 -1.9091557 15.15023 1.330787e-05 0.02710643 3.792988
## cg03167951 3.861347 -1.8898050 14.31875 1.789180e-05 0.02710643 3.572580
## cg25264081 3.609029 -1.1321228 13.87816 2.107141e-05 0.02710643 3.446994
## cg26684946 3.436919 -1.0220823 13.54920 2.388634e-05 0.02710643 3.348944
## cg20912169 4.109873 -0.2089213 13.54189 2.395385e-05 0.02710643 3.346719
## cg12859211 4.203347 -0.9553543 13.23941 2.695237e-05 0.02710643 3.253076
## cg13938881 3.330727 -1.3089052 13.06243 2.891236e-05 0.02710643 3.196712
## cg16935295 3.750777 -1.4133100 12.93627 3.041275e-05 0.02710643 3.155802
## cg23405575 3.469136 -0.2988488 12.75085 3.278844e-05 0.02710643 3.094543
## cg11855526 4.971912 -1.2052800 12.64822 3.419752e-05 0.02710643 3.060044

Note that since we performed our analysis on M-values, the logFC and AveExpr columns are computed on the M-value scale. For interpretability and visualisation we can look at the \(\beta\) values. The beta values for the top 4 differentially methylated CpGs shown in Figure 3.

cpgs <- rownames(top)
par(mfrow=c(2,2))
for(i in 1:4){
stripchart(beta[rownames(beta)==cpgs[i],]~design[,4],method="jitter",
group.names=c("Normal","Cancer"),pch=16,cex=1.5,col=c(4,2),ylab="Beta values",
vertical=TRUE,cex.axis=1.5,cex.lab=1.5)
title(cpgs[i],cex.main=1.5)
}

Figure 3: Top DM CpGs
The beta values for the top 4 differentially methylated CpGs.

7 Removing unwanted variation when testing for differential methylation

Like other platforms, 450k array studies are subject to unwanted technical variation such as batch effects and other, often unknown, sources of variation. The adverse effects of unwanted variation have been extensively documented in gene expression array studies and have been shown to be able to both reduce power to detect true differences and to increase the number of false discoveries. As such, when it is apparent that data is significantly affected by unwanted variation, it is advisable to perform an adjustment to mitigate its effects.

missMethyl provides a limma inspired interface to functions from the CRAN package ruv, which enable the removal of unwanted variation when performing a differential methylation analysis (Maksimovic et al. 2015).

RUVfit uses the RUV-inverse method by default, as this does not require the user to specify a \(k\) parameter. The ridged version of RUV-inverse is also available by setting method = rinv. The RUV-2 and RUV-4 functions can also be used by setting method = ruv2 or method = ruv4, respectively, and specifying an appropriate value for k (number of components of unwanted variation to remove) where \(0 \leq k < no. samples\).

All of the methods rely on negative control features to accurately estimate the components of unwanted variation. Negative control features are probes/genes/etc. that are known a priori to not truly be associated with the biological factor of interest, but are affected by unwanted variation. For example, in a microarray gene expression study, these could be house-keeping genes or a set of spike-in controls. Negative control features are extensively discussed in Gagnon-Bartsch and Speed (2012) and Gagnon-Bartsch et al. (2013). Once the unwanted factors are accurately estimated from the data, they are adjusted for in the linear model that describes the differential analysis.

If the negative control features are not known a priori, they can be identified empirically. This can be achieved via a 2-stage approach, RUVm. Stage 1 involves performing a differential methylation analysis using RUV-inverse (by default) and the 613 Illumina negative controls (INCs) as negative control features. This will produce a list of CpGs ranked by p-value according to their level of association with the factor of interest. This list can then be used to identify a set of empirical control probes (ECPs), which will capture more of the unwanted variation than using the INCs alone. ECPs are selected by designating a proportion of the CpGs least associated with the factor of interest as negative control features; this can be done based on either an FDR cut-off or by taking a fixed percentage of probes from the bottom of the ranked list. Stage 2 involves performing a second differential methylation analysis on the original data using RUV-inverse (by default) and the ECPs. For simplicity, we are ignoring the paired nature of the cancer and normal samples in this example.

# get M-values for ALL probes
meth <- getMeth(mSet)
unmeth <- getUnmeth(mSet)
M <- log2((meth + 100)/(unmeth + 100))

# setup design matrix
grp <- factor(targets$status,levels=c("normal","cancer"))
des <- model.matrix(~grp)
des

##   (Intercept) grpcancer
## 1           1         0
## 2           1         0
## 3           1         1
## 4           1         1
## 5           1         0
## 6           1         1
## attr(,"assign")
## [1] 0 1
## attr(,"contrasts")
## attr(,"contrasts")$grp
## [1] "contr.treatment"

# extract Illumina negative control data
INCs <- getINCs(rgSet)
head(INCs)

##          5723646052_R02C02 5723646052_R04C01 5723646052_R05C02
## 13792480        -0.3299654        -1.0955482        -0.5266103
## 69649505        -1.0354488        -1.4943396        -1.0067050
## 34772371        -1.1286422        -0.2995603        -0.8192636
## 28715352        -0.5553373        -0.7599489        -0.7186973
## 74737439        -1.1169178        -0.8656399        -0.6429681
## 33730459        -0.7714684        -0.5622424        -0.7724825
##          5723646053_R04C02 5723646053_R05C02 5723646053_R06C02
## 13792480        -0.6374299         -1.116598        -0.4332793
## 69649505        -0.8854881         -1.586679        -0.9217329
## 34772371        -0.6895514         -1.161155        -0.6186795
## 28715352        -1.7903619         -1.348105        -1.0067259
## 74737439        -0.8872082         -1.064986        -0.9841833
## 33730459        -1.5623138         -2.079184        -1.0445246

# add negative control data to M-values
Mc <- rbind(M,INCs)
# create vector marking negative controls in data matrix
ctl1 <- rownames(Mc) %in% rownames(INCs)
table(ctl1)

## ctl1
##  FALSE   TRUE 
## 485512    613

rfit1 <- RUVfit(data=Mc, design=des, coef=2, ctl=ctl1) # Stage 1 analysis
rfit2 <- RUVadj(rfit1)

Now that we have performed an initial differential methylation analysis to rank the CpGs with respect to their association with the factor of interest, we can designate the CpGs that are least associated with the factor of interest based on FDR-adjusted p-value as ECPs.

top1 <- topRUV(rfit2, num=Inf)
head(top1)

##                  X1       X1           X1      p.BH     p.ebayes p.ebayes.BH
## cg04743961 4.838190 26.74467 3.812882e-05 0.1401969 3.516091e-07  0.01017357
## cg07155336 5.887409 17.62103 1.608653e-04 0.1401969 3.583107e-07  0.01017357
## cg20925841 4.790211 26.69524 3.837354e-05 0.1401969 3.730375e-07  0.01017357
## cg03607359 4.394397 34.74068 1.542013e-05 0.1401969 4.721205e-07  0.01017357
## cg10566121 4.787914 21.80693 7.717708e-05 0.1401969 5.238865e-07  0.01017357
## cg07655636 4.571758 22.99708 6.424216e-05 0.1401969 6.080091e-07  0.01017357

ctl2 <- rownames(M) %in% rownames(top1[top1$p.ebayes.BH > 0.5,])
table(ctl2)

## ctl2
##  FALSE   TRUE 
## 172540 312972

We can then use the ECPs to perform a second differential methylation with RUV-inverse, which is adjusted for the unwanted variation estimated from the data.

# Perform RUV adjustment and fit
rfit3 <- RUVfit(data=M, design=des, coef=2, ctl=ctl2) # Stage 2 analysis
rfit4 <- RUVadj(rfit3)

# Look at table of top results
topRUV(rfit4)

##                  X1        X1          X1      p.BH     p.ebayes
## cg07155336 5.769286 15.345069 0.002005546 0.3431163 1.434834e-55
## cg06463958 5.733093 15.434797 0.001978272 0.3431163 6.749298e-55
## cg00024472 5.662959 15.946200 0.001832444 0.3431163 1.319390e-53
## cg02040433 5.651399 10.054445 0.005389436 0.3431163 2.146210e-53
## cg13355248 5.595396  9.963702 0.005504213 0.3431163 2.234891e-52
## cg02467990 5.592707  6.859614 0.013008521 0.3431163 2.499534e-52
## cg00817367 5.527501 13.070583 0.002921656 0.3431163 3.710480e-51
## cg11396157 5.487992 10.931263 0.004436178 0.3431163 1.873636e-50
## cg16306898 5.466780  5.573935 0.020790127 0.3431163 4.448085e-50
## cg03735888 5.396242 15.482605 0.001963955 0.3431163 7.700032e-49
##             p.ebayes.BH
## cg07155336 6.966293e-50
## cg06463958 1.638433e-49
## cg00024472 2.135266e-48
## cg02040433 2.605027e-48
## cg13355248 2.022589e-47
## cg02467990 2.022589e-47
## cg00817367 2.573547e-46
## cg11396157 1.137091e-45
## cg16306898 2.399554e-45
## cg03735888 3.738458e-44

Note, at present RUVm does not support contrasts, so only one factor of interest can be interrogated at a time using a design matrix with an intercept term.

# get M-values for ALL probes
meth <- getMeth(mSet)
unmeth <- getUnmeth(mSet)
M <- log2((meth + 100)/(unmeth + 100))

# setup design matrix
grp <- factor(targets$status,levels=c("normal","cancer"))
des <- model.matrix(~grp)
des

##   (Intercept) grpcancer
## 1           1         0
## 2           1         0
## 3           1         1
## 4           1         1
## 5           1         0
## 6           1         1
## attr(,"assign")
## [1] 0 1
## attr(,"contrasts")
## attr(,"contrasts")$grp
## [1] "contr.treatment"

# limma differential methylation analysis
lfit1 <- lmFit(M, design=des)
lfit2 <- eBayes(lfit1) # Stage 1 analysis

# Look at table of top results
topTable(lfit2)

## Removing intercept from test coefficients

##               logFC   AveExpr        t      P.Value   adj.P.Val        B
## cg07155336 6.037439 -1.276764 19.22210 1.175108e-07 0.005755968 7.635736
## cg04743961 4.887986 -2.317315 19.21709 1.177367e-07 0.005755968 7.634494
## cg03607359 4.393946 -2.191871 18.07007 1.852304e-07 0.005755968 7.334032
## cg13272280 4.559707 -2.099665 17.25531 2.599766e-07 0.005755968 7.099628
## cg22263007 4.438420 -1.010994 17.12384 2.749857e-07 0.005755968 7.060036
## cg03556069 5.456754 -1.811718 17.00720 2.891269e-07 0.005755968 7.024476
## cg08443814 4.597347 -2.062275 16.80835 3.151706e-07 0.005755968 6.962907
## cg18672939 5.159383 -0.705992 16.65643 3.368597e-07 0.005755968 6.915046
## cg24385334 4.157473 -1.943370 16.59313 3.463909e-07 0.005755968 6.894890
## cg18044663 4.426118 -1.197724 16.57851 3.486357e-07 0.005755968 6.890216

topl1 <- topTable(lfit2, num=Inf)

## Removing intercept from test coefficients

head(topl1)

##               logFC   AveExpr        t      P.Value   adj.P.Val        B
## cg07155336 6.037439 -1.276764 19.22210 1.175108e-07 0.005755968 7.635736
## cg04743961 4.887986 -2.317315 19.21709 1.177367e-07 0.005755968 7.634494
## cg03607359 4.393946 -2.191871 18.07007 1.852304e-07 0.005755968 7.334032
## cg13272280 4.559707 -2.099665 17.25531 2.599766e-07 0.005755968 7.099628
## cg22263007 4.438420 -1.010994 17.12384 2.749857e-07 0.005755968 7.060036
## cg03556069 5.456754 -1.811718 17.00720 2.891269e-07 0.005755968 7.024476

ctl3 <- rownames(M) %in% rownames(topl1[topl1$adj.P.Val > 0.5,])
table(ctl3)

## ctl3
##  FALSE   TRUE 
## 199150 286362

# Perform RUV adjustment and fit
rfit5 <- RUVfit(data=M, design=des, coef=2, ctl=ctl3) # Stage 2 analysis
rfit6 <- RUVadj(rfit5)

# Look at table of top results
topRUV(rfit6)

##                  X1        X1          X1      p.BH     p.ebayes
## cg06463958 5.910598 16.764282 0.001658268 0.2879799 7.201871e-67
## cg07155336 5.909549 16.280730 0.001775943 0.2879799 7.594724e-67
## cg02467990 5.841079  7.509890 0.010693351 0.2879799 2.388032e-65
## cg00024472 5.823529 16.922147 0.001622249 0.2879799 5.742351e-65
## cg01893212 5.699627  6.496989 0.014865392 0.2879799 2.611317e-62
## cg11396157 5.699331 11.927112 0.003673561 0.2879799 2.649365e-62
## cg13355248 5.658606 10.663862 0.004765630 0.2879799 1.924338e-61
## cg00817367 5.649284 14.067684 0.002499571 0.2879799 3.023717e-61
## cg16306898 5.610118  6.246593 0.016244793 0.2879799 2.002510e-60
## cg16556906 5.567659  6.525888 0.014716909 0.2879799 1.531939e-59
##             p.ebayes.BH
## cg06463958 1.843665e-61
## cg07155336 1.843665e-61
## cg02467990 3.864727e-60
## cg00024472 6.969951e-60
## cg01893212 2.143831e-57
## cg11396157 2.143831e-57
## cg13355248 1.334699e-56
## cg00817367 1.835064e-56
## cg16306898 1.080270e-55
## cg16556906 7.437748e-55

7.2 Visualising the effect of RUVm adjustment

To visualise the effect that the RUVm adjustment is having on the data, using an MDS plot for example, the getAdjusted function can be used to extract the adjusted values from the RUVm fit object. NOTE: The adjusted values should only be used for plotting - it is NOT recommended that they are used in any downstream analysis.

Madj <- getAdjusted(M, rfit6) # get adjusted values

The MDS plots below show how the relationship between the samples changes with and without RUVm adjustment. RUVm reduces the distance between the samples in each group by removing unwanted variation. It can be useful to examine this type of plot when trying to decide on the best set of ECPs or to help select the optimal value of \(k\), if using RUV-4 or RUV-2.

par(mfrow=c(1,2))
plotMDS(Mval, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Unadjusted")
legend("topleft",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

plotMDS(Madj, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Adjusted: RUV-inverse")
legend("topleft",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

Figure 4: RUVm adjusted data
An MDS plot of cancer and normal data, before and after RUVm adjustment.

To illustrate how the getAdjusted function can be used to help select an appropriate value for \(k\), we will run the second stage of the RUVm analysis using RUV-4 with two different \(k\) values.

# Use RUV-4 in stage 2 of RUVm with k=1 and k=2
rfit7 <- RUVfit(data=M, design=des, coef=2, ctl=ctl3,
                method = "ruv4", k=1) # Stage 2 with RUV-4, k=1
rfit8 <- RUVadj(rfit7)

rfit9 <- RUVfit(data=M, design=des, coef=2, ctl=ctl3,
                method = "ruv4", k=2) # Stage 2 with RUV-4, k=2
rfit10 <- RUVadj(rfit9)

# get adjusted values
Madj1 <- getAdjusted(M, rfit8)
Madj2 <- getAdjusted(M, rfit10)

The following MDS plots show how the relationship between the samples changes from the unadjusted data to data adjusted with RUV-inverse and RUV-4 with two different \(k\) values.

par(mfrow=c(2,2))
plotMDS(Mval, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Unadjusted")
legend("topleft",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

plotMDS(Madj, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Adjusted: RUV-inverse")
legend("topleft",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

plotMDS(Madj1, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Adjusted: RUV-4, k=1")
legend("bottomleft",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

plotMDS(Madj2, labels=targets$Sample_Name, col=as.integer(factor(targets$status)),
        main="Adjusted: RUV-4, k=2")
legend("bottomright",legend=c("Cancer","Normal"),pch=16,cex=1,col=1:2)

Figure 5: Effect of different adjustment methods and parameters
MDS plots of cancer and normal data before an after adjustment with RUV-inverse and RUV-4 with different k values.

8 Testing for differential variability (DiffVar)

8.1 Methylation data

Rather than testing for differences in mean methylation, we may be interested in testing for differences between group variances. For example, it has been hypothesised that highly variable CpGs in cancer are important for tumour progression (Hansen et al. 2011). Hence we may be interested in CpG sites that are consistently methylated in the normal samples, but variably methylated in the cancer samples.

In general we recommend at least 10 samples in each group for accurate variance estimation, however for the purpose of this vignette we perform the analysis on 3 vs 3. In this example, we are interested in testing for differential variability in the cancer versus normal group. Note that when we specify the coef parameter, which corresponds to the columns of the design matrix to be used for testing differential variability, we need to specify both the intercept and the fourth column. The ID variable is a nuisance parameter and not used when obtaining the absolute deviations, however it can be included in the linear modelling step. For methylation data, the function will take either a matrix of M-values, \(\beta\) values or a object as input. If \(\beta\) values are supplied, a logit transformation is performed. Note that as a default, varFit uses the robust setting in the limma framework, which requires the use of the statmod package.

fitvar <- varFit(Mval, design = design, coef = c(1,4))

The numbers of hyper-variable (1) and hypo-variable (-1) genes in cancer vs normal can be obtained using decideTests.

summary(decideTests(fitvar))

##        (Intercept) idid2 idid3 groupcancer
## Down             0     2     5           1
## NotSig       19713 19996 19957       19976
## Up             287     2    38          23

topDV <- topVar(fitvar, coef=4)
topDV

##            SampleVar LogVarRatio DiffLevene         t      P.Value
## cg19677203  6.894498    6.782612   3.736430  8.318855 9.159124e-17
## cg05495949  5.184611    3.686322   2.838095  6.029268 1.661438e-09
## cg17823751  5.505290    4.669976   3.010442  5.993857 2.066720e-09
## cg09491991  5.230551    6.307426   2.951419  5.216234 1.835053e-07
## cg05873889  7.305152    3.546095   2.626257  5.183382 2.189550e-07
## cg27047283  5.874682   -4.120722  -2.706340 -5.177864 2.255251e-07
## cg27018185  7.471801    2.571968   2.334681  5.134711 2.838993e-07
## cg16361302  3.991449    6.943687   2.884795  4.798362 1.605446e-06
## cg12521353  4.428729    2.818345   2.312976  4.741481 2.128912e-06
## cg22587602  5.507303    2.079294   1.964989  4.593123 4.379903e-06
##             Adj.P.Value
## cg19677203 1.831825e-12
## cg05495949 1.377814e-05
## cg17823751 1.377814e-05
## cg09491991 7.517505e-04
## cg05873889 7.517505e-04
## cg27047283 7.517505e-04
## cg27018185 8.111408e-04
## cg16361302 4.013615e-03
## cg12521353 4.730916e-03
## cg22587602 8.617455e-03

An alternate parameterisation of the design matrix that does not include an intercept term can also be used, and specific contrasts tested with contrasts.varFit. Here we specify the design matrix such that the first two columns correspond to the normal and cancer groups, respectively.

design2 <- model.matrix(~0+group+id)
fitvar.contr <- varFit(Mval, design=design2, coef=c(1,2))
contr <- makeContrasts(groupcancer-groupnormal,levels=colnames(design2))
fitvar.contr <- contrasts.varFit(fitvar.contr,contrasts=contr)

The results are identical to before.

summary(decideTests(fitvar.contr))

##        groupcancer - groupnormal
## Down                           1
## NotSig                     19976
## Up                            23

topVar(fitvar.contr,coef=1)

##            SampleVar LogVarRatio DiffLevene         t      P.Value
## cg19677203  6.894498    6.782612   3.736430  8.318855 9.159124e-17
## cg05495949  5.184611    3.686322   2.838095  6.029268 1.661438e-09
## cg17823751  5.505290    4.669976   3.010442  5.993857 2.066720e-09
## cg09491991  5.230551    6.307426   2.951419  5.216234 1.835053e-07
## cg05873889  7.305152    3.546095   2.626257  5.183382 2.189550e-07
## cg27047283  5.874682   -4.120722  -2.706340 -5.177864 2.255251e-07
## cg27018185  7.471801    2.571968   2.334681  5.134711 2.838993e-07
## cg16361302  3.991449    6.943687   2.884795  4.798362 1.605446e-06
## cg12521353  4.428729    2.818345   2.312976  4.741481 2.128912e-06
## cg22587602  5.507303    2.079294   1.964989  4.593123 4.379903e-06
##             Adj.P.Value
## cg19677203 1.831825e-12
## cg05495949 1.377814e-05
## cg17823751 1.377814e-05
## cg09491991 7.517505e-04
## cg05873889 7.517505e-04
## cg27047283 7.517505e-04
## cg27018185 8.111408e-04
## cg16361302 4.013615e-03
## cg12521353 4.730916e-03
## cg22587602 8.617455e-03

The \(\beta\) values for the top 4 differentially variable CpGs can be seen in Figure 6.

cpgsDV <- rownames(topDV)
par(mfrow=c(2,2))
for(i in 1:4){
stripchart(beta[rownames(beta)==cpgsDV[i],]~design[,4],method="jitter",
group.names=c("Normal","Cancer"),pch=16,cex=1.5,col=c(4,2),ylab="Beta values",
vertical=TRUE,cex.axis=1.5,cex.lab=1.5)
title(cpgsDV[i],cex.main=1.5)
}

Figure 6: Top DV CpGs
The beta values for the top 4 differentially variable CpGs.

8.2 RNA-Seq expression data

Testing for differential variability in expression data is straightforward if the technology is gene expression microarrays. The matrix of expression values can be supplied directly to the varFit function. For RNA-Seq data, the mean-variance relationship that occurs in count data needs to be taken into account. In order to deal with this issue, we apply a voom transformation (Law et al. 2014) to obtain observation weights, which are then used in the linear modelling step. For RNA-Seq data, the varFit function will take a DGElist object as input.

To demonstrate this, we use data from the tweeDEseqCountData package. This data is part of the International HapMap project, consisting of RNA-Seq profiles from 69 unrelated Nigerian individuals (Pickrell et al. 2010). The only covariate is gender, so we can look at differentially variable expression between males and females. We follow the code from the limma vignette to read in and process the data before testing for differential variability.

First we load up the data and extract the relevant information.

library(tweeDEseqCountData)
data(pickrell1)
counts<-exprs(pickrell1.eset)
dim(counts)

## [1] 38415    69

gender <- pickrell1.eset$gender
table(gender)

## gender
## female   male 
##     40     29

rm(pickrell1.eset)
data(genderGenes)
data(annotEnsembl63)
annot <- annotEnsembl63[,c("Symbol","Chr")]
rm(annotEnsembl63)

We now have the counts, gender of each sample and annotation (gene symbol and chromosome) for each Ensemble gene. We can form a DGElist object using the edgeR package.

library(edgeR)
y <- DGEList(counts=counts, genes=annot[rownames(counts),])

We filter out lowly expressed genes by keeping genes with at least 1 count per million reads in at least 20 samples, as well as genes that have defined annotation. Finally we perform scaling normalisation.

isexpr <- rowSums(cpm(y)>1) >= 20
hasannot <- rowSums(is.na(y$genes))==0
y <- y[isexpr & hasannot,,keep.lib.sizes=FALSE]
dim(y)

## [1] 17310    69

y <- calcNormFactors(y)

We set up the design matrix and test for differential variability. In this case there are no nuisance parameters, so coef does not need to be explicitly specified.

design.hapmap <- model.matrix(~gender)
fitvar.hapmap <- varFit(y, design = design.hapmap)

## Converting counts to log counts-per-million using voom.

fitvar.hapmap$genes <- y$genes

We can display the results of the test:

summary(decideTests(fitvar.hapmap))

##        (Intercept) gendermale
## Down             0          2
## NotSig           0      17308
## Up           17310          0

topDV.hapmap <- topVar(fitvar.hapmap,coef=ncol(design.hapmap))
topDV.hapmap

##                       Symbol Chr  SampleVar LogVarRatio DiffLevene         t
## ENSG00000213318 RP11-331F4.1  16 5.69839463   -2.562939 -0.9859943 -8.031243
## ENSG00000129824       RPS4Y1   Y 2.32497726   -2.087025 -0.4585620 -4.957005
## ENSG00000233864       TTTY15   Y 6.79004140   -2.245369 -0.6085233 -4.612934
## ENSG00000176171        BNIP3  10 0.41317384    1.199292  0.3632133  4.219404
## ENSG00000197358      BNIP3P1  14 0.39969125    1.149754  0.3353288  4.058147
## ENSG00000025039        RRAGD   6 0.91837213    1.091229  0.4926839  3.977022
## ENSG00000103671        TRIP4  15 0.07456448   -1.457139 -0.1520583 -3.911300
## ENSG00000171100         MTM1   X 0.44049558   -1.133295 -0.3334619 -3.896490
## ENSG00000149476          DAK  11 0.13289523   -1.470460 -0.1919880 -3.785893
## ENSG00000064886       CHI3L2   1 2.70234584    1.468059  0.8449434  3.782010
##                      P.Value  Adj.P.Value
## ENSG00000213318 7.238039e-12 1.252905e-07
## ENSG00000129824 3.960855e-06 3.428120e-02
## ENSG00000233864 1.496237e-05 8.633290e-02
## ENSG00000176171 6.441668e-05 2.787632e-01
## ENSG00000197358 1.147886e-04 3.973982e-01
## ENSG00000025039 1.527695e-04 4.375736e-01
## ENSG00000103671 1.921104e-04 4.375736e-01
## ENSG00000171100 2.022293e-04 4.375736e-01
## ENSG00000149476 2.956364e-04 4.425050e-01
## ENSG00000064886 2.995692e-04 4.425050e-01

The log counts per million for the top 4 differentially variable genes can be seen in Figure 7.

genesDV <- rownames(topDV.hapmap)
par(mfrow=c(2,2))
for(i in 1:4){
stripchart(cpm(y,log=TRUE)[rownames(y)==genesDV[i],]~design.hapmap[,ncol(design.hapmap)],method="jitter",
group.names=c("Female","Male"),pch=16,cex=1.5,col=c(4,2),ylab="Log counts per million",
vertical=TRUE,cex.axis=1.5,cex.lab=1.5)
title(genesDV[i],cex.main=1.5)
}

Figure 7: Top DV CpGs
The log counts per million for the top 4 differentially variably expressed genes.

9 Gene ontology analysis

Once a differential methylation or differential variability analysis has been performed, it may be of interest to know which gene pathways are targeted by the significant CpG sites. It is not entirely clear from the literature how best to perform such an analysis, however Geeleher et al. (Geeleher et al. 2013) showed there is a severe bias when performing gene ontology analysis with methylation data. This is due to the fact that there are differing numbers of probes per gene on several different array technologies. For the Illumina Infinium HumanMethylation450 array the number of probes per gene ranges from 1 to 1299, with a median of 15 probes per gene. For the EPIC array, the range is 1 to 1487, with a median of 20 probes per gene. This means that when mapping CpG sites to genes, a gene is more likely to be selected if there are many CpG sites associated with the gene.

One way to take into account this selection bias is to model the relationship between the number of probes per gene and the probability of being selected. This can be performed by adapting the goseq method of Young et al. (Young et al. 2010). Each gene then has a prior probability associated with it, and a modified version of a hypergeometric test can be performed, testing for over-representation of the selected genes in each gene set.

The gometh function performs gene set testing on GO categories or KEGG pathways (Phipson, Maksimovic, and Oshlack 2016). The gsameth function is a more generalised gene set testing function which can take as input a list of user specified gene sets. Note that for gsameth, the format for the gene ids for each gene in the gene set needs to be Entrez Gene IDs. For example, the entire curated gene set list (C2) from the Broad’s Molecular Signatures Database can be specified as input. The R version of these lists can be downloaded from http://bioinf.wehi.edu.au/software/MSigDB/index.html. Both functions take a vector of significant CpG probe names as input.

To illustrate how to use gometh, consider the results from the differential methylation analysis with RUVm.

topRUV(rfit4)

##                  X1        X1          X1      p.BH     p.ebayes
## cg07155336 5.769286 15.345069 0.002005546 0.3431163 1.434834e-55
## cg06463958 5.733093 15.434797 0.001978272 0.3431163 6.749298e-55
## cg00024472 5.662959 15.946200 0.001832444 0.3431163 1.319390e-53
## cg02040433 5.651399 10.054445 0.005389436 0.3431163 2.146210e-53
## cg13355248 5.595396  9.963702 0.005504213 0.3431163 2.234891e-52
## cg02467990 5.592707  6.859614 0.013008521 0.3431163 2.499534e-52
## cg00817367 5.527501 13.070583 0.002921656 0.3431163 3.710480e-51
## cg11396157 5.487992 10.931263 0.004436178 0.3431163 1.873636e-50
## cg16306898 5.466780  5.573935 0.020790127 0.3431163 4.448085e-50
## cg03735888 5.396242 15.482605 0.001963955 0.3431163 7.700032e-49
##             p.ebayes.BH
## cg07155336 6.966293e-50
## cg06463958 1.638433e-49
## cg00024472 2.135266e-48
## cg02040433 2.605027e-48
## cg13355248 2.022589e-47
## cg02467990 2.022589e-47
## cg00817367 2.573547e-46
## cg11396157 1.137091e-45
## cg16306898 2.399554e-45
## cg03735888 3.738458e-44

table(rfit4$p.ebayes.BH < 0.01)

## 
##  FALSE   TRUE 
## 424168  61344

At a 1% false discovery rate cut-off, there are still tens of thousands of CpG sites differentially methylated. These will undoubtably map to almost all the genes in the genome, making a gene ontology analysis irrelevant. One option for selecting CpGs in this context is to apply not only a false discovery rate cut-off, but also a \(\Delta\beta\) cut-off. However, for this dataset, taking a relatively large \(\Delta\beta\) cut-off of 0.25 still leaves more than 30000 CpGs differentially methylated.

beta <- getBeta(mSet)
beta_norm <- rowMeans(beta[,des[,2]==0])
beta_can <- rowMeans(beta[,des[,2]==1])
Delta_beta <- beta_can - beta_norm
sigDM <- rfit4$p.ebayes.BH < 0.01 & abs(Delta_beta) > 0.25
table(sigDM)

## sigDM
##  FALSE   TRUE 
## 451760  33748

Instead, we take the top 10000 CpG sites as input to gometh.

topCpGs<-topRUV(rfit4,number=10000)
sigCpGs <- rownames(topCpGs)
sigCpGs[1:10]

##  [1] "cg07155336" "cg06463958" "cg00024472" "cg02040433" "cg13355248"
##  [6] "cg02467990" "cg00817367" "cg11396157" "cg16306898" "cg03735888"

The takes as input a character vector of CpG names, and optionally, a character vector of all CpG sites tested. If the all.cpg argument is omitted, all the CpGs on the array are used as background. To change the array type, the array.type argument can be specified as either “450K” or “EPIC”. The default is “450K”.

If the plot.bias argument is TRUE, a figure showing the relationship between the probability of being selected and the number of probes per gene will be displayed.

For testing of GO terms, the collection argument takes the value “GO”, which is the default setting. For KEGG pathway analysis, set collection to “KEGG”. The function topGO shows the top enriched GO categories. For KEGG testing, use the topKEGG function. The functions goana and kegga are called by for GO and KEGG pathway analysis respectively.

For GO testing on our example dataset:

library(IlluminaHumanMethylation450kanno.ilmn12.hg19)
gst <- gometh(sig.cpg=sigCpGs, all.cpg=rownames(rfit2), collection="GO")

## Warning in alias2SymbolTable(flat$symbol): Multiple symbols ignored for one
## or more aliases

topGO(gst)

##                                              Term Ont    N   DE         P.DE
## GO:0048731                     system development  BP 4699  938 4.262167e-34
## GO:0048856       anatomical structure development  BP 5710 1080 5.424830e-34
## GO:0007399             nervous system development  BP 2271  573 1.651669e-33
## GO:0007275     multicellular organism development  BP 5240 1009 5.295109e-33
## GO:0032502                  developmental process  BP 6093 1123 4.212913e-32
## GO:0031226 intrinsic component of plasma membrane  CC 1664  397 9.147205e-31
## GO:0032501       multicellular organismal process  BP 7302 1239 2.115364e-30
## GO:0005887  integral component of plasma membrane  CC 1587  378 2.936688e-29
## GO:0005886                        plasma membrane  CC 5129  927 2.206206e-28
## GO:0030154                   cell differentiation  BP 4021  799 3.076791e-28
## GO:0071944                         cell periphery  CC 5236  941 1.214878e-27
## GO:0048869         cellular developmental process  BP 4201  819 7.442595e-27
## GO:0030182                 neuron differentiation  BP 1303  367 8.301010e-27
## GO:0009653     anatomical structure morphogenesis  BP 2579  587 1.119083e-26
## GO:0007417     central nervous system development  BP  939  283 2.225010e-26
## GO:0022008                           neurogenesis  BP 1540  412 2.389448e-26
## GO:0048699                  generation of neurons  BP 1444  393 4.200997e-26
## GO:0044459                   plasma membrane part  CC 2723  572 9.110275e-25
## GO:0031224        intrinsic component of membrane  CC 5186  856 2.662385e-23
## GO:0007267                    cell-cell signaling  BP 1576  383 6.557603e-23
##                     FDR
## GO:0048731 6.172643e-30
## GO:0048856 6.172643e-30
## GO:0007399 1.252901e-29
## GO:0007275 3.012520e-29
## GO:0032502 1.917465e-28
## GO:0031226 3.469382e-27
## GO:0032501 6.877048e-27
## GO:0005887 8.353776e-26
## GO:0005886 5.578513e-25
## GO:0030154 7.001854e-25
## GO:0071944 2.513363e-24
## GO:0048869 1.411426e-23
## GO:0030182 1.453124e-23
## GO:0009653 1.819070e-23
## GO:0007417 3.375637e-23
## GO:0022008 3.398542e-23
## GO:0048699 5.623653e-23
## GO:0044459 1.151792e-21
## GO:0031224 3.188837e-20
## GO:0007267 7.461569e-20

For a more generalised version of gene set testing in methylation data where the user can specify the gene set to be tested, the function gsameth can be used. To display the top 20 pathways, topGSA can be called. gsameth can take a single gene set, or a list of gene sets. The gene identifiers in the gene set must be Entrez Gene IDs. To demonstrate gsameth, a toy example is shown below, with gene sets made up of randomly selected genes from the org.Hs.eg.db package.

library(org.Hs.eg.db)

## Loading required package: AnnotationDbi

genes <- toTable(org.Hs.egSYMBOL2EG)
set1 <- sample(genes$gene_id,size=80)
set2 <- sample(genes$gene_id,size=100)
set3 <- sample(genes$gene_id,size=30)
genesets <- list(set1,set2,set3)
gsa <- gsameth(sig.cpg=sigCpGs, all.cpg=rownames(rfit4), collection=genesets)

## Warning in alias2SymbolTable(flat$symbol): Multiple symbols ignored for one
## or more aliases

topGSA(gsa)

##        N DE       P.DE       FDR
## [1,]  30  2 0.06610367 0.1983110
## [2,]  80  2 0.55560923 0.8334138
## [3,] 100  1 0.84061499 0.8406150

Note that if it is of interest to obtain the Entrez Gene IDs that the significant CpGs are mapped to, the getMappedEntrezIDs can be called.

10 Session information

sessionInfo()

R version 3.5.1 Patched (2018-07-12 r74967) Platform: x86_64-pc-linux-gnu (64-bit) Running under: Ubuntu 16.04.5 LTS

Matrix products: default BLAS: /home/biocbuild/bbs-3.8-bioc/R/lib/libRblas.so LAPACK: /home/biocbuild/bbs-3.8-bioc/R/lib/libRlapack.so

locale: [1] LC_CTYPE=en_US.UTF-8 LC_NUMERIC=C
[3] LC_TIME=en_US.UTF-8 LC_COLLATE=C
[5] LC_MONETARY=en_US.UTF-8 LC_MESSAGES=en_US.UTF-8
[7] LC_PAPER=en_US.UTF-8 LC_NAME=C
[9] LC_ADDRESS=C LC_TELEPHONE=C
[11] LC_MEASUREMENT=en_US.UTF-8 LC_IDENTIFICATION=C

attached base packages: [1] stats4 parallel stats graphics grDevices utils datasets [8] methods base

other attached packages: [1] org.Hs.eg.db_3.7.0
[2] AnnotationDbi_1.44.0
[3] edgeR_3.24.0
[4] tweeDEseqCountData_1.19.0
[5] minfiData_0.27.0
[6] IlluminaHumanMethylation450kanno.ilmn12.hg19_0.6.0 [7] IlluminaHumanMethylation450kmanifest_0.4.0
[8] minfi_1.28.0
[9] bumphunter_1.24.0
[10] locfit_1.5-9.1
[11] iterators_1.0.10
[12] foreach_1.4.4
[13] Biostrings_2.50.0
[14] XVector_0.22.0
[15] SummarizedExperiment_1.12.0
[16] DelayedArray_0.8.0
[17] BiocParallel_1.16.0
[18] matrixStats_0.54.0
[19] Biobase_2.42.0
[20] GenomicRanges_1.34.0
[21] GenomeInfoDb_1.18.0
[22] IRanges_2.16.0
[23] S4Vectors_0.20.0
[24] BiocGenerics_0.28.0
[25] limma_3.38.0
[26] missMethyl_1.16.0
[27] BiocStyle_2.10.0

loaded via a namespace (and not attached): [1] colorspace_1.3-2
[2] siggenes_1.56.0
[3] mclust_5.4.1
[4] rprojroot_1.3-2
[5] base64_2.0
[6] bit64_0.9-7
[7] xml2_1.2.0
[8] codetools_0.2-15
[9] splines_3.5.1
[10] methylumi_2.28.0
[11] knitr_1.20
[12] Rsamtools_1.34.0
[13] IlluminaHumanMethylationEPICanno.ilm10b4.hg19_0.6.0 [14] annotate_1.60.0
[15] GO.db_3.7.0
[16] HDF5Array_1.10.0
[17] BiocManager_1.30.3
[18] readr_1.1.1
[19] compiler_3.5.1
[20] httr_1.3.1
[21] backports_1.1.2
[22] lazyeval_0.2.1
[23] assertthat_0.2.0
[24] Matrix_1.2-14
[25] htmltools_0.3.6
[26] prettyunits_1.0.2
[27] tools_3.5.1
[28] bindrcpp_0.2.2
[29] gtable_0.2.0
[30] glue_1.3.0
[31] GenomeInfoDbData_1.2.0
[32] dplyr_0.7.7
[33] doRNG_1.7.1
[34] Rcpp_0.12.19
[35] multtest_2.38.0
[36] preprocessCore_1.44.0
[37] nlme_3.1-137
[38] rtracklayer_1.42.0
[39] DelayedMatrixStats_1.4.0
[40] xfun_0.4
[41] stringr_1.3.1
[42] rngtools_1.3.1
[43] IlluminaHumanMethylationEPICmanifest_0.3.0
[44] statmod_1.4.30
[45] XML_3.98-1.16
[46] beanplot_1.2
[47] scales_1.0.0
[48] zlibbioc_1.28.0
[49] MASS_7.3-51
[50] hms_0.4.2
[51] rhdf5_2.26.0
[52] GEOquery_2.50.0
[53] RColorBrewer_1.1-2
[54] yaml_2.2.0
[55] gridExtra_2.3
[56] memoise_1.1.0
[57] ggplot2_3.1.0
[58] pkgmaker_0.27
[59] biomaRt_2.38.0
[60] reshape_0.8.8
[61] stringi_1.2.4
[62] RSQLite_2.1.1
[63] highr_0.7
[64] genefilter_1.64.0
[65] GenomicFeatures_1.34.0
[66] bibtex_0.4.2
[67] rlang_0.3.0.1
[68] pkgconfig_2.0.2
[69] bitops_1.0-6
[70] nor1mix_1.2-3
[71] evaluate_0.12
[72] lattice_0.20-35
[73] purrr_0.2.5
[74] Rhdf5lib_1.4.0
[75] bindr_0.1.1
[76] ruv_0.9.7
[77] GenomicAlignments_1.18.0
[78] bit_1.1-14
[79] tidyselect_0.2.5
[80] plyr_1.8.4
[81] magrittr_1.5
[82] bookdown_0.7
[83] R6_2.3.0
[84] DBI_1.0.0
[85] pillar_1.3.0
[86] withr_2.1.2
[87] survival_2.43-1
[88] RCurl_1.95-4.11
[89] tibble_1.4.2
[90] crayon_1.3.4
[91] rmarkdown_1.10
[92] progress_1.2.0
[93] grid_3.5.1
[94] data.table_1.11.8
[95] blob_1.1.1
[96] digest_0.6.18
[97] xtable_1.8-3
[98] tidyr_0.8.2
[99] illuminaio_0.24.0
[100] munsell_0.5.0
[101] openssl_1.0.2
[102] registry_0.5
[103] BiasedUrn_1.07
[104] quadprog_1.5-5