LawlessGenomics

Summary - inference of causal metabolite networks

Sun, 13 Aug 2023 00:00:00 +0200

Summary - inference of causal metabolite networks

13 Aug 2023

A summary of: Chen, S., Lin, Z., Shen, X., Li, L., & Pan, W. (2023). Inference of causal metabolite networks in the presence of invalid instrumental variables with GWAS summary data. Genetic Epidemiology, 1–15, https://doi.org/10.1002/gepi.22535, (Chen et al., n.d.).

Briefly, this paper is about using instrumental variables (SNPs) in causal inference with applications to genome-wide association studies (GWAS).

Exposure:
- Metabolites: They serve as intermediate phenotypes connecting genetic variants to clinical outcomes and play a crucial role in biological processes.
Instrumental Variable (IV):
- SNPs (Single Nucleotide Polymorphisms): These are used as instrumental variables to isolate the variability in metabolites that is independent of confounders.
Outcome:
- Clinical outcomes: These are affected by metabolites, which in turn are influenced by SNPs. The causal effect of metabolites on clinical outcomes is what the study aims to determine.

For a more gradual background jump down to the “In context” section.

Abstract summary

Structural equation models (SEMs) for inferring causal networks in metabolites and other complex traits. The method:

Performs causal analysis to discover relationships among multiple traits.
Accounts for potential invalid IVs.
Allows for data analysis using only GWAS summary statistics.
Considers bidirectional relationships between traits.

The approach uses a stepwise selection to identify invalid IVs, and demonstrates its superior performance using both real and simulated GWAS data.

Key methods summary

2.1 One-sample data

For one-sample GWAS individual-level data:

For each of $n$ individuals:
- We have $p$ SNPs as IVs and $M$ traits.
- Traits are denoted by an $n x M$ matrix $Y = [y_1, y_2, ... , y_M]$.
  - Where $y_i = (y_1i, y_2i, ..., y_ni)^T$ is the vector of $n$ observations for trait $i = 1, 2, ..., M$.
- IVs are denoted as $Z = [z_1, ... , z_p]$.
  - Where $z_j = (z_1j, ..., z_nj)^T$ is the vector of the $n$ observations for IV $j = 1, 2, ..., p$.

2.2 SEM with individual-level data

SEM is used as a tool for multivariate causal inference in this framework.
The analysis is based on one-sample GWAS individual-level data.
The system considers linear equations for the ‘n’ individuals in the sample.
Notations follow Wang et al. 2016.
The model has random errors denoted by a ‘n x M’ matrix.
The matrix ‘E’ consists of vectors for the ‘n’ random errors for each trait from 1 to M.
The assumption is made that the expected value of error ‘e_i’ is 0.
For each trait, the error follows a normal distribution with mean 0 and variance ‘σ_i^2’.
The matrix ‘M x M’ denotes the coefficients for the traits, represented by ‘Γ’.
The coefficients for each trait ‘i’ range from 1 to M.
The linear model for the ‘i-th’ trait is represented by an equation where traits are connected linearly and the sum is 0.

2.3 SEM with some invalid IVs

Discusses the scenario when some IVs may be invalid.
Violation of one or more of the three valid IV Assumptions (A1)–(A3).
Proposes a method to account for invalid IVs in SEM.
For trait i, the first ${p}_{0i}$ IVs are considered invalid IVs.
Represents the matrix of invalid IVs with ${Z}_{ {\mathscr{A}}_{i}}=[{z}_{1},{\rm{\ldots }},{z}_{p_{0i}}]$.
${z}_{j}={({z}_{1j},{\rm{\ldots }},{z}_{nj})}^{T}$ is the vector for the n observations of IV $j=1,2,{\rm{\ldots }},{p}_{0i}$.
$M$ vectors are present for the coefficients of invalid IVs: ${B}_{1},{\rm{\ldots }},{B}_{M}$.
${B}_{i}={({\beta }_{1i},{\beta }_{2i},{\rm{\ldots }},{\beta }_{p_{0i}i})}^{T}$ represents the direct or (horizontal) pleiotropic effects of the invalid IVs on trait $i=1,2,{\rm{\ldots }},M$.
The linear SEM for trait i is represented as: ${y}_{1}{\gamma }_{1i}+\cdots +{y}_{i}{\gamma }_{ii}+\cdots +{y}_{M}{\gamma }_{Mi}+{z}_{1}{\beta }_{1i}+\cdots +{z}_{p_{0}i}{\beta }_{p_{0i}i}+{e}_{i}=0,$.
$\gamma$’s and $\beta$’s are unknown parameters in the model.

Code

onesample mvstepIV

The original source code is here: https://github.com/chen-siyi7/one-sample-stepwise-IV-selection/blob/main/one-sample%20stepwise%20IV%20code.R

The function onesample_mvstepIV conducts one-sample stepwise IV.

Input Parameters:

p: Total number of predictors.
R: Correlation matrix.
betaZX: Regression coefficients for predictors.
betaZY: Regression coefficients for outcomes.
se_betaZY: Standard error of betaZY.
n: Sample size.
gamma_hat: Gamma hat values (prior information).

Main Computations:

Initialize ZTZ using R as: $ZTZ = R$
Compute ZTY as the element-wise product of ZTZ diagonal and betaZY: $ZTY = \text{diag}(ZTZ) \times \beta{ZY}$
Calculate the median of YTY for each predictor SNP as: $YTY[SNP] = (n-1) \times ZTZ[SNP,SNP] \times (se_\beta{ZY}^2)[SNP] + ZTY[SNP] \times \beta{ZY}[SNP]$, excluding NA values.
Compute Bayesian Information Criterion (BIC) for each predictor. For each predictor i:
- a. Create a matrix test11 with diagonal element i set to 1.
- b. Create matrix W1 by combining test11 and gamma_hat.
- c. Solve for W1 using: $\text{solve.W1} = W1^T \times ZTZ \times W1$
- d. Compute beta estimates beta1 as: $\beta1 = (solve.W1^{-1} \times W1^T \times ZTY)$
- e. Calculate BIC as: $testbic[i] = n \times \log(YTY - \beta1^T \times W1^T \times ZTY) + \log(n) \times \sum_{i} \text{diag}(test11)$
Determine the optimal instrument variables (IVs) based on BIC:
- a. For each iteration j, select the predictor i with the smallest BIC.
- b. Repeat the process by adding one predictor at a time.
- c. Stop if the current and previous IV are the same.
Compute the final beta estimates using the invalid IVs:
- a. Extract the invalid IVs from whichIV and set their diagonal elements in test11 to 1.
- b. Compute beta estimates beta1 as: $\beta1 = (solve.W1^{-1} \times W1^T \times ZTY)$
- c. Calculate variance of beta Varbeta as: $\text{Var\beta} = \text{diag}(solve.W1 \times n) \times \sigma_u2$ where $\sigma_u2 = YTY - \beta1^T \times W1^T \times ZTY$.

Output:

invalidIV: Indices of invalid IVs.
beta_est: Estimated beta values.
beta_se: Standard error of beta estimates.
K: Number of invalid IVs.

onesample mvstepIV ind

The original source code is here: https://github.com/chen-siyi7/one-sample-stepwise-IV-selection/blob/main/onesample_mvstepIV_ind%20code.R

The function onesample_mvstepIV_ind performs one-sample stepwise IV for independent SNPs.

Input Parameters:

Y: Response variable.
Z: Predictor matrix.
n: Sample size.
gamma_hat: Gamma hat values (prior information).

Main Computations:

Initialize testbic for Bayesian Information Criterion.
For each predictor i:
- a. Initialize a zero vector l with length dim(Z)[2] and set the ith element to 1.
- b. Modify matrix Z22 such that for each row j, Z22[j,] is Z[j,]*l.
- c. Perform a linear regression (lm_stage2) of Y on Z22 and Z*gamma_hat.
- d. Calculate BIC for this predictor using: $testbic[i] = n \times \log\left(\frac{\sum(lm\_stage2\text{residuals}^2)}{n}\right) + \log(n) \times \sum(l)$
Determine the optimal instrument variables (IVs) based on BIC:
- a. For each iteration j, select the predictor i with the smallest BIC.
- b. Modify matrix Z22 for the selected predictors and add one predictor at a time.
- c. Repeat the linear regression and calculate BIC as in step 2.
- d. Stop if the current and previous IV are the same.
Extract the invalid IVs, which.invalid, from whichIV and sort them to obtain K.
Compute the final beta estimates using the invalid IVs:
- a. Extract columns K from Z to form Z22.
- b. Perform linear regression (lm_stage2) of Y on Z22 and Z*gamma_hat.
- c. Calculate beta estimates betaest as: $\beta{est} = \text{summary}(lm\_stage2)\text{coef[,1]}$
- d. Calculate variance of residuals sigma_u2 as: $\sigma_u2 = \frac{\sum(lm\_stage2\text{residuals}^2)}{n}$
- e. Calculate variance of beta Varbeta using: $\text{Varbeta} = \text{diag}(ginv(X^TX)) \times \sigma_u2$ where $X = \text{cbind}(Z22, Dhat)$ and $Dhat = Z*gamma\_hat$.
- f. Compute the standard error betase as: $\beta{se} = \sqrt{\text{Varbeta}}$

Output:

beta_est: Estimated beta values.
beta_se: Standard error of beta estimates.
invalid IVs: Indices of invalid IVs.
no. of invalid IV: Number of invalid IVs.

In context

Recap:

Exposure:
- Metabolites: They serve as intermediate phenotypes connecting genetic variants to clinical outcomes and play a crucial role in biological processes.
Instrumental Variable (IV):
- SNPs (Single Nucleotide Polymorphisms): These are used as instrumental variables to isolate the variability in metabolites that is independent of confounders.
Outcome:
- Clinical outcomes: These are affected by metabolites, which in turn are influenced by SNPs. The causal effect of metabolites on clinical outcomes is what the study aims to determine.

Background

In GWAS, associations are generally sought between single nucleotide polymorphisms (SNPs) and a single trait. But GWAS data can also be used to analyze multiple related traits, leading to improved power and new biological insights. Specifically, network analysis of multiple traits is gaining interest, especially when it comes to causal network analysis. This is pivotal for elucidating relationships among multiple traits, such as in gene network and protein network analyses. Metabolite network analysis, the focal point of this research, posits that metabolites are integral parts of many biological processes, often interacting with each other in regulatory networks. By inferring these networks, we can gain insight into relationships among metabolites in biological processes.

In causal networks, traits, including metabolites, proteins, and genes, serve as the nodes. Their causal relationships are represented by directed edges connecting them. SNPs are utilized as instrumental variables (IVs). To model these intricate biological networks, structural equation models (SEMs) have been adopted.

What is an Instrumental Variable (IV)?

An instrumental variable is associated with the exposure but does not have a direct association with the outcome, except through its relationship with the exposure. Its role is to isolate the variability in the exposure that is independent of the confounders.

Key Assumptions

Relevance: The IV is correlated with the exposure.
Exclusion: The IV only affects the outcome through its effect on the exposure.
Exchangeability: The IV is not associated with unobserved confounders.

How Does It Work?

IV analysis uses the variation in the exposure explained by the instrument to estimate the causal effect of the exposure on the outcome.

Why is it Necessary for Causal Inference?

Control for Unmeasured Confounding: IVs can provide unbiased estimates of causal effects when unmeasured or unobserved confounding is present.
Endogeneity: IVs can solve the problem of endogeneity.
Natural Experiments: IVs can be employed in “natural experiments” where random assignment of treatments is not feasible.

Usage

In the context of GWAS and metabolite network analysis, IV methods are crucial. They help determine causal relationships in complex biological processes, especially when metabolites, which do not function in isolation, interact within metabolite regulatory networks.

Limitations

Weak Instruments: Weak correlation between the IV and exposure can lead to biased IV estimates.
Violations of Assumptions: IV estimates can be biased if any core assumptions are violated.
Interpretability: The causal effect estimated through IV is often specific to a particular population, reducing generalizability.

Summary

Instrumental Variables are a pivotal tool in causal inference, especially in genome-wide association studies (GWAS). When utilized properly, they can provide valuable insights into causal relationships in settings laden with confounding and endogeneity. However, they come with their own assumptions and potential limitations.

References

Chen, S., Lin, Z., Shen, X., Li, L., & Pan, W. Inference of causal metabolite networks in the presence of invalid instrumental variables with GWAS summary data. Genetic Epidemiology, n/a(n/a). https://doi.org/https://doi.org/10.1002/gepi.22535

What I want from genomic data centers

Tue, 09 May 2023 00:00:00 +0200

What I want from genomic data centers

09 May 2023

No one asked, but here are some things that would make me love a genomic data center.

Sensible file naming: Require a certain ID format before submission. Ensure that the sequencing center and customers use consistent file naming conventions. Sequencing typical requires output like: SAMPLEID_MACHINE_RUNID. However, if a customer provides a sample ID like “ABC-001”, the sequencing center should NOT be returning “ABC_001_XYZ0001_001234”. Avoid discrepancies between the customer’s original ID and the sequencing center’s modified ID. If modifications are required, ensure that the customer does first it on their side by submitting “ABC_001”.
Protocol file: Provide a summary file that lists the number of samples, batch IDs, sample handling steps (e.g. library preparation kit used, sequencing lane, sequencer machine used), and any alignment or reference genome information. The customer should be responsible for recording what they have ordered but we must assume that they will loose it if it is not included in the data pack which they are returned.
Alignment: If you can provide it as an option, align the data to the most commonly accepted reference genome, such as GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz. For more refer to Heng Li and Illumina. Assume that the customer has no idea which to use unless they specific it. Include a note in the protocol file with advice on alternative reference genomes, such as:

If you map reads to GRCh37 or hg19, use hs37-1kg: ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/human_g1k_v37.fasta.gz. If you map to GRCh37 and believe decoy sequences help with better variant calling, use hs37d5: ftp://ftp-trace.ncbi.nih.gov/1000genomes/ftp/technical/reference/phase2_reference_assembly_sequence/hs37d5.fa.gz. If you map reads to GRCh38 or hg38, use the following: ftp://ftp.ncbi.nlm.nih.gov/genomes/all/GCA/000/001/405/GCA_000001405.15_GRCh38/seqs_for_alignment_pipelines.ucsc_ids/GCA_000001405.15_GRCh38_no_alt_analysis_set.fna.gz.
Variant calling: If providing variant calling services, such as using GATK best-practices, include a summary of the protocol.
Reporting prioritization: Focus on implementing the most important reporting methods first, such as:

Producing a simple report will be valuable for the majority of customers. Avoid reporting overly complex information.

AI - From Complex Requests to Simple Fixes

Wed, 18 Jan 2023 01:00:01 +0100

AI - From Complex Requests to Simple Fixes

18 Jan 2023

Edit 2026: This post was written two months after the public launch of OpenAI’s chatGPT (Nov 2021). It seems rather basic now but it is a small reminder of the turning point and shows how blown away we were with this new tech.

I see a lot of people giving their opinions about chat AI without much examples of real-world application. So here is my concrete experience. In two weeks I have rewritten, from scratch, one of my genomics pipelines which can analyse 1000 clinical exomes is a few hours.

It included:

42 scripts/programs.
3694 lines of code.
Several languages.

It works to:

Process raw fastq up to germline variant calling.
Using high-performance computing cluster.
Process vcf into clinical variant interpretation.
Options for custom filtering strategies.
Statistical analysis and logs.
Replaced several mainstream tools that are difficult to manage.

Final results:

My work was ~6-10x times faster.
I probably would not have rewritten this better version as it would have taken too long.
Instead of getting stuck at any difficult impasse I can get context-specific alternatives and testable example code.

I use chatGPT plus as a paid member for access without restriction. Instead of googling for keywords that match, I either post a code example and ask for specific fixes or simply describe the request. Here is a very simple example, but I have done some very complex request with great success.

Example:

I have an R dataframe “df” with columns “genotype” “variant” “gene” “sample” “inheritance_model”.

Add a new column “flag”.

Group by “gene” and “sample”,

then add the value 1 to “flag” for rows where “genotype” > 1 and “inheritance_model” == AR.

Result:

library(dplyr)

df <- df %>%
  group_by(gene, sample) %>%
  mutate(flag = ifelse(genotype > 1 & inheritance_model == "AR", 1, 0))

My real-world use is usually far more complex than this and uses relatively large sets of code. However, my queries are broken up or pre-empted with an introductory discussion such that, in context, the final request is as simple as the one above.

Final thought

Whatever sales reps or armchair philosophers say about the rapidly approaching era of AI, if you start using these tools now you will rapidly understand how powerful they will become in many aspects of life.

(ChatGPT also picked this nice blogpost title for me when I fed it this blogpost and asked for a suggestion).

AI, tell me about precision medicine genomics

Wed, 18 Jan 2023 01:00:01 +0100

AI, tell me about precision medicine genomics

18 Jan 2023

Tell me about precision medicine genomics.
Tell me how AI can be used in this context.
In this context, is AI better than classical statistical analysis and machine learning?
Summarize all of the previous answers into one paragraph.
Thank you.

The following is a conversation with ChatGPT.

Tell me about precision medicine genomics.

“Precision medicine genomics is a medical approach that uses genetic information to tailor medical treatment to the individual patient. This approach is based on the understanding that genetic variations can affect a person’s response to certain drugs and susceptibility to certain diseases. By identifying these genetic variations, doctors can select the most effective treatment options for each patient and potentially avoid harmful side effects.

“One important aspect of precision medicine genomics is the use of genetic testing to identify specific mutations or variations in a patient’s DNA. This information can be used to guide treatment decisions, such as selecting the most appropriate drug or dosage. For example, if a patient has a specific genetic mutation that is known to cause resistance to a certain drug, the doctor may choose to prescribe a different drug instead.

“Another important aspect of precision medicine genomics is the use of large-scale genomic data to identify new targets for drug development and to develop new diagnostic tests. By analyzing the genomic data of large numbers of patients, researchers can identify genetic variations that are associated with specific diseases or drug responses. This information can be used to develop new drugs or diagnostic tests that are tailored to the specific genetic makeup of individual patients.

“Precision medicine genomics is still in its early stages and there are still many challenges to be overcome, such as the high cost of genetic testing and the need for more research to fully understand the implications of genetic variations on disease and drug response. However, as the field continues to evolve, it has the potential to revolutionize the way medicine is practiced, leading to more effective and personalized treatments for patients.”

Tell me how AI can be used in this context.

“Artificial intelligence (AI) can be used in various ways to support precision medicine genomics. Some examples include:

Genetic variant analysis: AI algorithms can be used to analyze large amounts of genomic data to identify genetic variations that are associated with specific diseases or drug responses. This can help researchers identify new targets for drug development and develop new diagnostic tests.
Drug discovery: AI can be used to analyze large amounts of data on drug interactions and side effects to identify new drug candidates that have a high likelihood of being effective and safe.
Predictive modeling: AI algorithms can be used to create predictive models that can identify patients who are most likely to respond to specific treatments based on their genetic profile. This can help doctors make more informed treatment decisions.
Personalized medicine: AI can be used to analyze a patient’s genetic data, medical history, and other relevant information to create a personalized treatment plan.
Natural Language Processing (NLP) : AI-based NLP can help in extracting insights from unstructured data sources such as electronic medical records, clinical notes, and scientific literature.
Imaging analysis : AI can be used to analyze medical images such as CT scans, MRI’s and X-rays to identify and classify disease patterns.

“AI is still in early stages of development in the field of precision medicine genomics, but it has the potential to revolutionize the field by making it possible to analyze and interpret large amounts of data quickly and accurately. This can help to improve the accuracy of diagnoses and treatment decisions, leading to better outcomes for patients.”

In this context, is AI better than classical statistical analysis and machine learning?

“In the context of precision medicine genomics, AI can be considered as an extension of classical statistical analysis and machine learning. It can be used to analyze and interpret large amounts of data quickly and accurately, which can make it a useful tool for precision medicine genomics. However, it is important to note that AI is not always better than classical statistical analysis and machine learning. The choice of method depends on the specific use case and the data being analyzed.

“AI can be particularly useful for analyzing large amounts of complex data, such as genomic data, which can be difficult to analyze using traditional methods. For example, AI can be used to identify patterns in the data that might not be apparent using classical statistical analysis. Additionally, AI can be used to create predictive models that can identify patients who are most likely to respond to specific treatments based on their genetic profile, which can be difficult to do using traditional methods.

“On the other hand, classical statistical analysis and machine learning still have their own advantages in specific scenarios. For example, traditional statistical analysis methods are well established and have been widely used in the field of medical research for decades. They are also relatively easy to understand and interpret. Additionally, classical statistical analysis is often more computationally efficient than AI, which can make it a better choice in certain cases.

“In summary, AI can be a valuable tool in precision medicine genomics, but it is not always better than classical statistical analysis and machine learning. The choice of method depends on the specific use case and the data being analyzed.”

Summarize all of the previous answers into one paragraph.

“Precision medicine genomics is a medical approach that uses genetic information to tailor medical treatment to the individual patient. Genetic testing can be used to identify specific mutations or variations in a patient’s DNA. Artificial intelligence (AI) can be used to analyze large amounts of genomic data to identify genetic variations that are associated with specific diseases or drug responses. AI can be used to analyze large amounts of data on drug interactions and side effects to identify new drug candidates that have a high likelihood of being effective and safe. AI can be used to create predictive models that can identify patients who are most likely to respond to specific treatments based on their genetic profile. However, it is important to note that AI is not always better than classical statistical analysis and machine learning, the choice of method depends on the specific use case and the data being analyzed.”

Thank you.

“You’re welcome! I’m glad I could help. Let me know if you have any other questions.”

We need a nice rare disease database

Thu, 30 Jun 2022 02:00:01 +0200

We need a nice rare disease database

30 June 2022

What is this?

This short post is to demonstrate a problem and a solution as an example of why I add these things to my public page. It is just one of the hundreds of small, modular datasets that I build and cross-link for all my public and private LawlessGenomics work. For precision medicine we need countless small, efficient, connected, tools in a consisten format. This is one example.

Problem

Diagnosing rare immune disorders is very challenging. A fast checklist of candidate genes is required.

One step to the solution

One of the most important resources for clinical immunology genetics is the yearly report of the Inborn Errors of Immunity (IEI) Committee by the International Union of Immunological Societies (IUIS) (https://iuis.org), (Tangye et al., 2022). Read the paper here: doi.org/10.1007/s10875-022-01289-3 and directly download their original xlsx table from Springer here: Supplementary file2 (XLSX 93 kb).

However

Because of traditional publishing protocols, there is a major problem. To use this dataset you must either:

Download and save an XLSX table (download).
Bookmark the webpage and view a png screen-shot of each table.
Download the pdf version and view an embedded pdf of each table.

Better solution

For years, I have made personal databases of gene lists which I query very often. Every bioinformatician also uses large sets of databases for annotation.

I notice a huge gap for clinicians, etc. who also need this info but have to rely on looking up the paper each time. One good example of the correct method is the Genomics England PanelApp https://panelapp.genomicsengland.co.uk.

I want a fast, searchable version of the IUIS IEI gene list. So I made it here:

Human Inborn Errors of Immunity.

Why is this better?

I don’t just want to confirm that my query gene is an immune disorder-related gene, I want to quickly get a lot of facts about it. Bioinformatically, I would add all this data in one step using my variant effect pipeline. However, I usually look this stuff up while on the go, in a meeting, etc. For a clinician, it is the same. They do not have the time to call up and request a bioinformatic query. We all need a fast, searchable table.

I have added additional hyperlinks to all of the external websites that are most pertinent. I also did a lot of cleaning and standardisation for filtering “Inheritance” terms, etc. so that it can work seamlessly.

Thanks

Many thanks to the committee for releasing their invaluable updates every year.

Please do not contact me to complain! I hope I have modified and cited the original information enough to deter a take-down request. I also hope someone adds a version to the IUIS website.

Anecdote

I was lucky enough to have a paper which accounts for one of these newly added genes, since this list is rapidly approaching its asymptote: Credited for a newly added immune gene in IUIS.

References

Tangye, S. G., Al-Herz, W., Bousfiha, A., Cunningham-Rundles, C., Franco, J. L., Holland, S. M., Klein, C., Morio, T., Oksenhendler, E., Picard, C., Puel, A., Puck, J., Seppänen, M. R. J., Somech, R., Su, H. C., Sullivan, K. E., Torgerson, T. R., & Meyts, I. (2022). Human Inborn Errors of Immunity: 2022 Update on the Classification from the International Union of Immunological Societies Expert Committee. Journal of Clinical Immunology. https://doi.org/10.1007/s10875-022-01289-3

KZNF protein pathway regulation

Fri, 24 Jun 2022 02:00:01 +0200

KZNF protein pathway regulation

24 June 2022

Exec summary
Background
- Typical approach
- New approach
Case study
- Protocol
- Result
Summary
Code used
References

Exec summary

We use existing data to show an improved method for identifying ZNF-dependent gene regulation at the:
1. gene level.
2. protein pathway level.
The method is statistically unbiased and might be used to produce more successful functional validation studies.

I will go thorough the steps to get to this final summary image:

Background

Control of gene expression by KZNFs occurs throughout the genome.
Clusters of several genes within a single region are affected by one KZNF binding site.
Research often focuses on one KZNF at a time
- functional experiment require a ZNF in vitro system and Chip-seq
Research often focuses on the resulting best candidate:
- a single genome site that had the larges change in expression
- to top gene in that region (for whatever reason)
- nearst gene to the regulotory region

ZNF genes that are intolerant to LoF (pLI = 1) are very interesting:
- since their function is regulation of expression for target genes
- the targets themselves must also have a intollerance to LoF
ZNF targets can be many gene (i) genome wide (ii) clustered together.
Genome-wide ZNF targets might form a protein pathway
pLI ZNF targets may also be (i) individually pLI or (ii) cumulatively pLI for a pathway.

Typical approach

Make a ZNF in vitro assay
Chip-seq
Find genes with DE
Explain the top hit
Or pass all hits into GO for a pathway report.

Opinion:

Result #4 is interesting and requires a lot of functional work.
Result #5 is generally not very insightful and often focuses on a cluster of genes, rather than genome-wide.

New approach

Select a pLI ZNF (pilot study)
Make a ZNF in vitro assay
Chip-seq
Find genes with DE
Cluster all target genes into their protein pathways
Identify pathways where all genes are (i) individually pLI or (ii) cumulatively pLI for the pathway.

Opinion:

This result unbiased for picking what you determine to be the “top hit”.
The mechanisms of pLI (and therefore ZFN downstream effect) can be interpreted from existing evidence
- for all genes withing the pathway, functional explanation is likely avaiable (Uniprot + GTEx)
A successful pilot in the most damaging pLI will provide a route to more complex and subtle regulatory mechanisms
- i.e. genes/pathways that are tollerant to haploinsuffiency which would otherwise be at the mercy of blind functional analysis.

Case study

Protocol

0. Select a pLI ZNF (pilot study)

The closer pLI is to one, the more intolerant of protein-truncating variants the transcript appears to be. We consider pLI ≥ 0.9 as an extremely intolerant set of transcripts. (Lek et al., 2016).

Methods in Code block 1

To find ZNF genes with pLI >0.9 I used

https://gnomad.broadinstitute.org/downloads
Section “Constraint”
File “pLoF Metrics by Gene TSV”
Ref https://www.nature.com/articles/s41586-020-2308-7
(Lek et al., 2016).

Handling requires installation of samtools https://www.htslib.org (Samtools, BCFtools, HTSlib). Uncompress the data with bgzip.

Methods in Code block 2

Query the whole genoe pLI score for all ZNF genes. Clean the dataset to extract the data of interest: Gene name, numbers of observed/expected LoF, pLI score, etc.

Methods in Code block 3

The R code performs a lot of formatting and cleaning to interpret the dataset. We find all genes wiht pLI > 0.9.

I have pre-emptively shown ZNF777 in red. I will show in the next section why I chose this as a pilot study.

1. Make a ZNF in vitro assay

This was found from existing data, as shown in the next section.

2. Chip-seq

This was found from existing data, as shown in the next section.

3. Find genes with DE

I used existing chip-seq data. This will pair one ZNF to a list of all genes that it ostensibly regulates. I believe this dataset to be a useful example: (Imbeault et al., 2017), https://www.nature.com/articles/nature21683.

“We then determined the genomic targets of human KZFPs by performing chromatin immunoprecipitation with exonuclease digestion (ChIP–exo)[13] on 257 HEK293T cell lines, each engineered to express one haemagglutinin (HA)-tagged family member. We obtained single base-pair resolution binding sites for 222 human KZFPs (Supplementary Table 4), 12 of them in duplicate (Extended Data Fig. 2), with the number of high-quality peaks per protein ranging from more than 10,000 to only around 15 (Extended Data Fig. 3a).

I compared my list of all ZNF for pLI >0.9 with the ZNF genes available in this study. The first match was ZNF777, which has pLI 0.99 (as shown in Figure 1).

I downloaded the data:

GSM2466654 ZNF777 -https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE78099 -https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM2466654
Methods in Code block 3

I cleaned and visualise the MACS (Model-based Analysis of ChIP-Seq) peak data. What is MACS? - ref 1, (Feng et al., 2011). What is MACS? - ref 2

4. Cluster all target genes into their protein pathways

Methods in Code block 4
Methods in Code block 5
Methods in Code block 6 (The data contained genomic coordinates so I needed to gene coding region protein for further steps.)
Methods in Code block 3

The genes “affected” by ZNF777 were clustered into their best-fit protein pathways using STRING-db. String uses experimental and curated data from Biocarta, BioCyc, GO, KEGG, and Reactome, BIND, DIP, GRID, HPRD, IntAct, MINT, and PID.

I access the database using Cytoscape.

Pathway clustering was performed using Markov Cluster Algorithm (MCL), an unsupervised cluster algorithm for networks based on stochastic flow. (Van Dongen, 2000), https://micans.org/mcl/.

In Figure A, I show all of the gene target of ZNF777.

Each node illustrates one protein.
Each edge illustrates the weight of evidence connecting proteins.

Since we cannot define the protein pathways clearly here (something that you find when reading hierarchical GO enrichement analsis results), I instead cluster these proteins into the best-fitting pathways.

The result of MCL clustering is show in Figure B.

Now we have a set of distinct protein pathways.

5. Identify pathways with pLI

Methods in Code block 3

Next, since we started with a ZNF that is intolerant to LoF, we also believe that the target genes must also be

(i) individually pLI
or (ii) cumulatively pLI for the pathway.

I cleaned the data further to group gene according to their

protein pathways,
genomic distribution, and
pLI score.

Figure 3 shows the top 5 largest pathways.

Key: We can see that rahter than focusing on a cluster of genes within one genomic region, we interpret genes according to their protein pthway.
Some protein pathways will have high pLI scores and may be considered the major ZNF targets.
Some protein pathways can tolerate LoF and maybe considered by-standers.

These by-stander genes are still surely sensitive to regulation by ZNF777. However, we can infer the evolutionary hierarchy of the major and minor protein pathways under the control of this ZNF.

6. Further development

There are 2 major further ways to progress.

Complete all avaialble Chip-seq datasets for pLI >0.9. Expand into pLI below 0.9 if possible.
Find which genes are in these protein pathways and see why they are not regulated by this ZNF (and see if there are sister ZNFs working in tangent to control pathway dosages).

Result

From all candidate ZNF gene targets, we find those which match it’s intolerance to LoF. We therefore define these pathways as the major targets of this ZNF.

I have purposely refrained for completing the final step: quantifying the top pLI pathway/gene list. This is to prevent from making a conclusion without testing the theory and stats.

Summary

Code used

1.unzip.sh

#!/bin/bash

# dataset
# <https://gnomad.broadinstitute.org/downloads>
# Section "Constraint"
# File "pLoF Metrics by Gene TSV"
# Ref <https://www.nature.com/articles/s41586-020-2308-7>

# install samtools

bgzip -d gnomad.v2.1.1.lof_metrics.by_gene.txt.bgz

2.scan.sh

#!/bin/bash
set -e
cd ../data/

output=ZNF_summary.tsv

# query gene and get summary column
head -1 gnomad.v2.1.1.lof_metrics.by_gene.txt | cut -f 1,17,18,20,21,42,43,46,75,76,77 > $output
# obs_lof	mu_lof	exp_lof	pLI	obs_het_lof	obs_hom_lof	exp_hom_lof

grep "ZNF*" gnomad.v2.1.1.lof_metrics.by_gene.txt | cut -f 1,17,18,20,21,42,43,46,75,76,77 >> $output

3.analyse.R

Major note: some of the data used in the second half of this script requires the scripts from 4,5,6. I have named them in the order of writing. It will be obvious when a dataset is missing, that the next bash script should be run. If continuing this project, 3.analyse.R should be split into sequential order, or call the bash script within the R code.

Minor note: This script is needlessly long and contains rough exploration work. If continuing this project, it can be heavily simplified.

library(dplyr)
library(ggplot2)
library(tidyr)

# Multiple sequence alignment file
df <- 
	read.table(file="../data/ZNF_summary.tsv", 
				  header = TRUE, 
				  sep = "\t", 
				  stringsAsFactors = FALSE)

# Note there is one KAZN gene included, figure out the grep correction later.

# The closer pLI is to one, the more intolerant of protein-truncating variants the transcript appears to be. We consider pLI ≥ 0.9 as an extremely intolerant set of transcripts. _Lek et al Nature 2016._
names(df)

# get the data in chr order
df <- arrange(df, chromosome, start_position)

df$chromosome <- as.numeric(df$chromosome)
df$start_position <- as.numeric(df$start_position)

# replace NAs with X chromosome 
df$chromosome[is.na(df$chromosome)]<-"X"

# create the subset for example: ZNF777
g1 <- subset(df, gene == "ZNF777")

pZNF777 <- 
	df %>%
#	ggplot(aes(x=start_position, y=pLI))+
	ggplot(aes(x=factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE)),
				  y=pLI))+
#	geom_point() +
	geom_jitter() +
	geom_point(data=g1, colour="red") +  # this adds a red point
	geom_text(data=g1, label="ZNF777", y=.95, colour="red") +
	geom_hline(linetype="dotted", yintercept=0.9) +
	#facet_grid(~chromosome) +
#	facet_grid(.~factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE))) + 
#	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
#			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
#			axis.text.x=element_blank()) +
	xlab("ZNF genes genomic position") +
	ylab("pLI (≥ 0.9 extremely intolerant to LoF)") 

pZNF777

# Next I want to simulate some chip-seq data, matching one ZNF to a list of all genes that are regulated
# I believe this dataset is useful
#### https://www.nature.com/articles/nature21683
# We then determined the genomic targets of human KZFPs by performing chromatin immunoprecipitation with exonuclease digestion (ChIP–exo)13 on 257 HEK293T cell lines, each engineered to express one haemagglutinin (HA)-tagged family member. We obtained single base-pair resolution binding sites for 222 human KZFPs (Supplementary Table 4), 12 of them in duplicate (Extended Data Fig. 2), with the number of high-quality peaks per protein ranging from more than 10,000 to only around 15 (Extended Data Fig. 3a). 

#ZNF77 has pLI 0.99
# Download
# GSM2466654	ZNF777
# https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSE78099
## https://www.ncbi.nlm.nih.gov/geo/query/acc.cgi?acc=GSM2466654

df2 <- 
	read.table(file="../data/GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.txt", 
				  header = FALSE, 
				  sep = "\t", 
				  stringsAsFactors = FALSE)

df2 <- separate(df2, V1, into = c("text", "chromosome"), sep = "chr") %>% select(-text)

colnames(df2)[colnames(df2) == 'V2'] <- 'start_position'
colnames(df2)[colnames(df2) == 'V3'] <- 'end_position'
colnames(df2)[colnames(df2) == 'V5'] <- 'score'

df2 %>%
	ggplot(aes(x=start_position, y=score ))+
	geom_point() +
#	geom_hline(linetype="dotted", yintercept=0.9) +
	#facet_grid(~chromosome) +
	facet_grid(.~factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE))) + 
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			axis.text.x=element_blank()) +
	xlab("Genes genomic position") +
	ylab("Score MACS peak") 

# aligned to GRCh37g1k 

# get gene symbtols via ensenbke biomart
# http://grch37.ensembl.org/biomart/martview/35883cadc275e8738b07668ab1f8d58f

# run 4.chip.sh

# Query for these coordinates and output attibutes:
#Dataset : Human genes (GRCh37.p13)
#Filters: coordinates, e.g. 1:100:10000:-1, 1:100000:200000:1: [ID-list specified]
#Attributes:
#Gene stable ID
#Gene start (bp)
#Gene end (bp)
#Chromosome/scaffold name
#Transcription start site (TSS)
#Gene name

# Output: GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.biomartformat_annotated.txt

df3 <- 
	read.table(file="../data/GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.biomartformat_annotated.txt", 
				  header = TRUE, 
				  sep = "\t", 
				  stringsAsFactors = FALSE)

# run 5.genelist.sh

# cytoscape 
# string gene query, print list
# MCL cluster, default 2.5 inflation

df4 <- 
	read.table(file="../output/img2_string_network_ZNF777_peaks_MCL2.5_table.csv", 
				  header = TRUE, 
				  sep = ",", 
				  stringsAsFactors = FALSE)

df4 <- df4 %>% select(X__mclCluster, display.name) 

# colnames(df)[colnames(df) == 'oldName'] <- 'newName'
colnames(df4)[colnames(df4) == 'display.name'] <- 'Gene.name'
colnames(df4)[colnames(df4) == 'X__mclCluster'] <- 'mclCluster'

# merge MCL clusters with the MACS peaks (genes named via biomart)
df5 <- merge(df3, df4)

# get first in gene (instead of all TSS)
df5 <- df5 %>%
	group_by(Gene.name) %>%
	filter(row_number()==1)

names(df5)

colnames(df5)[colnames(df5) == 'Chromosome.scaffold.name'] <- 'chromosome'
colnames(df5)[colnames(df5) == 'Gene.start..bp.'] <- 'start_position'
colnames(df5)[colnames(df5) == 'Gene.end..bp.'] <- 'end_position'

df5 %>%
	ggplot(aes(x=start_position, y=(mclCluster), color= (as.character(mclCluster) ) ))+
	geom_point() +
	#	geom_hline(linetype="dotted", yintercept=0.9) +
	#facet_grid(~chromosome) +
	facet_grid(~factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE))) + 
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			axis.text.x=element_blank()) +
	xlab("Genes genomic position") +
	ylab("Score MACS peak") 


df5 %>%
	ggplot(aes(x=factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE)), 
				  y=(mclCluster), 
				  group=mclCluster,
				  color= (as.character(mclCluster) ) ))+
	#geom_point() +
	geom_jitter() +
	geom_line(aes(alpha=0.2)) +
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			#axis.text.x=element_blank()
			) +
	xlab("Genes genomic position") +
	ylab("Protein pathway ID") 

# merge protein pathway data with MACS peaks

df5 <- df5 %>% filter(mclCluster < 6)

df2
names(df2)
names(df5)

tmp <- merge(
	(df2 %>% select(chromosome, start_position, end_position, score)),
	(df5 %>% select(Gene.name, chromosome, start_position, end_position, mclCluster)),
	all = T )

df2 <- df2 %>% 
	select(chromosome, start_position, end_position, score)

tmp <- merge(df2, df5, all = T )

# get pLI scores
# now get the plof for mcl genes
gene_names <- tmp %>% filter(mclCluster > 0) %>% select(Gene.name)

write.table(gene_names, file='../output/gene_names_mcl_top5.csv',sep=",", quote=FALSE, row.names=FALSE, col.names = FALSE)

# run 6.mclplof.sh

# Multiple sequence alignment file
df6 <- 
	read.table(file="../output/gene_names_mcl_top5_plof.txt", 
				  header = TRUE, 
				  sep = "\t", 
				  stringsAsFactors = FALSE)

names(tmp)
names(df6)

# colnames(df6)[colnames(df6) == 'oldName'] <- 'newName'
colnames(df6)[colnames(df6) == 'gene'] <- 'Gene.name'

tmp <- merge(tmp, df6, all.x = T)

tmp %>%
	ggplot(aes(x=factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE)), 
				  y=mclCluster, 
				  group=mclCluster,
				  color= (as.character(mclCluster) ) ))+
#	geom_point() +
	geom_jitter() +
	geom_line(aes(alpha=0.2)) +
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			#axis.text.x=element_blank()
	) +
	xlab("Genes genomic position") +
	ylab("Protein pathway ID") 

p0 <- tmp %>%
#	filter(chromosome ==1 ) %>%
	ggplot(aes(x=start_position, 
				  y=mclCluster, 
				  group=mclCluster,
				  color= (as.character(mclCluster) ) ))+
#	geom_point() +
	geom_jitter() +
	geom_line(aes(alpha=0.2)) +
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			#axis.text.x=element_blank()
	) +
	xlab("Genes genomic position") +
	ylab("Protein pathway ID") 

g2$Gene.name
g2 <- subset(tmp, pLI >= 0.9)
p1 <- tmp2 %>%
	ggplot(aes(x=factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE)), 
				  y=(mclCluster), 
				  group=mclCluster,
				  color= (as.character(mclCluster) ) ))+
	#geom_point() +
	geom_jitter(aes(size=pLI)) +
	geom_line(aes(alpha=0.2)) +
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			#axis.text.x=element_blank()
	) +
	xlab("Genes genomic position") +
	ylab("Protein pathway ID")  + 
	geom_text(data=g2, aes(label=Gene.name), color='black')

p2 <- tmp %>%
#	filter(chromosome ==1 ) %>%
	ggplot(aes(start_position, score ))+
	geom_segment(aes(xend = start_position, yend = 0))+
	facet_grid(.~factor(chromosome, stringr::str_sort(unique(chromosome), numeric = TRUE))) +
	theme(legend.position="none", panel.background = element_rect("#F7F7F7"),
			#axis.text.x=element_text(angle=45,hjust=1,vjust=0.5),
			axis.text.x=element_blank()) +
	xlab("Genes genomic position") +
	ylab("Score MACS peak") 

# Plot main figure ----
require(gridExtra)
grid.arrange(pZNF777, p2, p1,ncol=1, 
				 bottom = "", 
				 left = "") 

4.chip.sh

#!/bin/bash

cd ../data/

# convert to correct format
# (Chr:Start:End:Strand) [Max 500 advised]
# e.g. 1:100:10000:-1, 1:100000:200000:1

cut -f 1-3 GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.txt |\
	sed 's/	/:/g' |\
	sed 's/chr//g' \
	> GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.biomartformat.txt

5.genelist.sh

#!/bin/bash

# gene simple gene list to cluster with STRING
cut -f 6 GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.biomartformat_annotated.txt |\
	sort |\
	uniq \
	> GSM2466654_ZNF777_peaks_processed_score_signal_exo.bed.biomartformat_annotated_genes.txt

6.mclplof.sh

#!/bin/bash
cd ../output/

# for every line in the top 5 protein pathways, get the pLI data
head -1 ../data/gnomad.v2.1.1.lof_metrics.by_gene.txt | cut -f 1,17,18,20,21,42,43,46,75,76,77 > gene_names_mcl_top5_plof.txt

while read -r line; 
do 
grep "$line"  ../data/gnomad.v2.1.1.lof_metrics.by_gene.txt;
done < gene_names_mcl_top5.csv  |\
	cut -f 1,17,18,20,21,42,43,46,75,76,77 >> gene_names_mcl_top5_plof.txt

References

Lek, M., Karczewski, K. J., Minikel, E. V., Samocha, K. E., Banks, E., Fennell, T., O’Donnell-Luria, A. H., Ware, J. S., Hill, A. J., Cummings, B. B., & others. (2016). Analysis of protein-coding genetic variation in 60,706 humans. Nature, 536(7616), 285.
Imbeault, M., Helleboid, P.-Y., & Trono, D. (2017). KRAB zinc-finger proteins contribute to the evolution of gene regulatory networks. Nature, 543(7646), 550–554.
Feng, J., Liu, T., & Zhang, Y. (2011). Using MACS to identify peaks from ChIP-Seq data. Current Protocols in Bioinformatics, 34(1), 2–14.
Van Dongen, S. M. (2000). Graph clustering by flow simulation [PhD thesis].

Reference genome choice

Mon, 31 Jan 2022 01:00:01 +0100

Reference genome choice

Reference genome choice is discussed succinctly in many difference places. Therefore, I will not write a long piece on it but instead add links for what I think are usefule sources.

Heng Li - Which human reference genome to use?

https://lh3.github.io/2017/11/13/which-human-reference-genome-to-use

Illumina review

https://www.illumina.com/science/genomics-research/articles/dragen-demystifying-reference-genomes.html

References

Who's afraid of the big bad geneticist?

Tue, 21 Sep 2021 00:00:00 +0200

Who’s afraid of the big bad geneticist?

Unfinished article

Understanding public perception of genomics is crucial for ethically establishing systems of translational medicine. Despite efforts towards effective altruism, the use and strategies of genomic intervention ought remain in the control of the public; the greatest tasking lies in adequate technical education for a topics with a high barrier to entry. Human genomics may be one field where oversimplified summaries are not sufficient to demonstrate the grand value for human health, agriculture, and other unexpected benefits. I shall continue to add high-quality resources that may be useful to this post, over time.

Public perception

In studying the “Global Public Perceptions of Genomic Data Sharing: What Shapes the Willingness to Donate DNA and Health Data?”, (Middleton et al., 2020) perform analysis of peoples’ familiarity with the concepts of DNA and genomics, and associations in the willingness to donate DNA and medical information. Publishor url: https://www.cell.com/ajhg/fulltext/S0002-9297(20)30292-5 and link to pdf.

The analysis have been adjusted for sociodemographic factors, etc. Not all demographies were homogeneously familiar with genetics, but the meta-analysis showed associations between understanding and participation willingness.

There was evidence for between-country heterogeneity ($I^2$ = 48% and $I^2$ = 61% for familiarity with genetics and personal experience, respectively).
Compared to participants who were unfamiliar with genetics, those who were familiar or had personal experience had greater odds of being willing to donate (odds ratio [OR] = 1.85, 95% CI = 1.11-2.00; OR = 2.70, 95% CI = 2.37- 3.09, respectively).
Similar results were seen when considering single groups of recipients of donated DNA and medical information, i.e. doctors, non-profit researchers, for-profit researchers.
The association between genetic exceptionalism (perception DNA is different from other medical data) and willingness to donate was OR 1.60 (95% CI = 1.47–1.75).
There was a strong association between trust in multiple actors (doctors, non-profit researchers, commercial researchers, governments, etc.) and willingness to donate DNA and medical information. The pooled OR was 3.85 (95% CI = 3.34–4.44).

To re-illustrate the authors’ work I have downloaded the data and reproduces the characteristics plots here as follows. This data was used for the meta-analysis to report the ORs listed above. The R code that I used for this process is included.

# Data source: DOI:https://doi.org/10.1016/j.ajhg.2020.08.023
# Download Supplemental data (mmc3.xlsx) Table S2. Unadjusted Results for Full Dataset, Stratified by Country
# Convert data to tsv format.

data <-
  read.table(file="mmc3.tsv",
             header = TRUE,
             sep = "\t")

require(tidyr)
data_l <- gather(data, cohort, value,
       Total:United.States, na.rm = FALSE, convert = FALSE, factor_key = FALSE)

data_l_sep <- separate(data_l, value,
                       sep = "\\(|\\)",
                            into = c("value", "samples"))

# Get variable names for filtering
unique(data_l_sep$Variable)

Then plotting the main figures as follows

require(ggplot2)
# Figure 1. Willingness to Donate Anonymous DNA and Medical Information to Different Recipient Groups, Stratified by Country
df1 <- data_l_sep %>%
  filter(Variable == "Willingness to donate to at least two recipient groups" |
           Variable == "Willingness to donate to non-profit researchers" |
           Variable == "Willingness to donate to for-profit researchers") %>%
  filter(Categories == "Yes") %>%
  group_by(Variable)

df1 %>%
ggplot(aes(x=factor(cohort), y=as.numeric(value), fill=factor(Variable))) +
  geom_bar(stat="identity", position="dodge")+
  theme(axis.text.x = element_text( angle = 45, hjust = 1 ),
        legend.position="bottom", legend.direction="vertical") +
  labs(x = "Cohort", y = " Percentage willingness ",
       title ="Figure 1. Willingness to Donate Anonymous DNA and\nMedical Information to Different Recipient Groups")

# Figure 2 Familiarity with Genetics (Including Familiarity Gained through Personal Experience), Stratified by Country
df2 <- data_l_sep %>%
  filter(Variable == "Familiarity with genetics") %>%
  filter(Categories == "Personal" | Categories == "Familiar" | Categories == "Unfamiliar") %>%
  group_by(Variable)

df2 %>%
  ggplot(aes(x=factor(cohort), y=as.numeric(value), fill=factor(Categories))) +
  geom_bar(stat="identity", position="dodge")+
  theme(axis.text.x = element_text( angle = 45, hjust = 1 ),
        legend.position="bottom", legend.direction="vertical") +
  labs(x = "Cohort", y = " Percentage",
       title ="Figure 2 Familiarity with Genetics \nIncluding Familiarity Gained through Personal Experience")

# Figure 3 Perception of DNA as Being Different from Other Medical Information (Genetic Exceptionalism) versus DNA as Being the Same (or Unsure), Stratified by Country
df3 <- data_l_sep %>%
  filter(Variable == "Perception of DNA being different to other medical information versus DNA being the same") %>%
  filter(Categories == "Same/Unsure" | Categories == "Different") %>%
  group_by(Variable)

df3 %>%
  ggplot(aes(x=factor(cohort), y=as.numeric(value), fill=factor(Categories))) +
  geom_bar(stat="identity", position="dodge")+
  theme(axis.text.x = element_text( angle = 45, hjust = 1 ),
        legend.position="bottom", legend.direction="horizontal") +
  labs(x = "Cohort", y = " Percentage",
       title ="Figure 3 Perception of DNA as Being Different from\nOther Medical Information (Genetic Exceptionalism)\nversus DNA as Being the Same (or Unsure)")

# Figure 4 Trust in Donating DNA and Medical Information to More than One User (Including Doctor, Researcher, Company, Government, etc.), Stratified by Country
df4 <- data_l_sep %>%
  filter(Variable == "Trust in donating DNA and medical information to more than one user") %>%
  group_by(Variable)

df4 %>%
  ggplot(aes(x=factor(cohort), y=as.numeric(value), fill=factor(Categories))) +
  geom_bar(stat="identity", position="dodge")+
  theme(axis.text.x = element_text( angle = 45, hjust = 1 ),
        legend.position="bottom", legend.direction="horizontal") +
  labs(x = "Cohort", y = " Percentage",
       title ="Figure 4 Trust in Donating DNA and Medical Information to \nMore than One User (incl. Doctor, Researcher, Government, etc.)")

References

Middleton, A., Milne, R., Almarri, M. A., Anwer, S., Atutornu, J., Baranova, E. E., Bevan, P., Cerezo, M., Cong, Y., Critchley, C., & others. (2020). Global public perceptions of genomic data sharing: what shapes the willingness to donate DNA and health data? The American Journal of Human Genetics, 107(4), 743–752.

Weeknotes

Mon, 13 Sep 2021 00:00:00 +0200

Weeknotes

13 Sept 2021

Work

Writing paper on viral infections.
Creating genome and gene domain structure illustrations with features in illustrator and bed, MSA, and fasta files.

Personal

Working on astro biology projects.
Learning a 2d/3d architecture system for design and rendering.

Reading

A Short History of the Middle East: From Ancient Empires to Islamic State by Gordon Kerr link
Killing Hope: U.S. Military and C.I.A. Interventions Since World War II by William Blum link - highly recommended.

Other media

Podcast - Afropop Worldwide The Cameroon Cuba connection.

Weeknotes

Mon, 06 Sep 2021 00:00:00 +0200

Weeknotes

6 Sept 2021

Work

Writing paper on viral infections.
Review grant.
Prep for incoming student projects.

Personal

Working on astro biology projects.

Reading

A Short History of the Middle East: From Ancient Empires to Islamic State by Gordon Kerr link
Smiley’s People by John le Carré - link.

Other media

Podcast - Afropop Worldwide New Moves in Afro-Jazz
Podcast - Afropop Worldwide Hip Deep Angola 4: The Cuban Intervention in Angola

Aggregated Cauchy Association Test (ACAT)

Tue, 25 Apr 2023 00:00:00 +0000

Aggregated Cauchy Association Test (ACAT)

Abbreviations
Intro to this topic
Papers
Step-by-step explanation of ACAT.R
Applying STAAR-O for multiple annotation weights
Non-gene-centric analysis using dynamic windows with SCANG-STAAR
Multi-weight annotation analysis
Main equations for ACAT
tan and $\pi$
Original R code from yaowuliu

25 Apr 2023 - last update

Abbreviations

ACAT: Aggregated Cauchy Association Test
ACAT-V: Aggregated Cauchy Association Test - Variant level
ACAT-O: Aggregated Cauchy Association Test - Omnibus
SKAT: Sequence Kernel Association Test
ARIC: Atherosclerosis Risk in Communities

Intro to this topic

The Aggregated Cauchy Association Test (ACAT) is a statistical method used for rare-variant association tests (RVATs) in genetic studies. ACAT is designed to aggregate the association signals of multiple rare genetic variants within a genomic region or a gene, while accounting for the directions of the effects of these variants on the phenotype of interest. The ACAT method utilizes a Cauchy distribution, which allows for improved performance in identifying true associations, especially when the directions and magnitudes of variant effects are heterogeneous.

First, here is a great talk by author of SKAT and other methods:

Watch on YouTube Dr. Xihong Lin: Overview of Rare Variant Analysis of Whole Genome Sequencing Association Studies.

The major part starts at time: 22:30
and the best is at: 30:50 where she describes the aggregated Cauchy association test (ACAT) method for combining multiple annotations (like CADD score, MAF, etc.) to calculate the final P-value.
This is their annotation database discussed: https://favor.genohub.org

Papers

ACAT paper, Yaowu Liu, et al AJGH 2019.
Application of STAAR protocol to TOPMed, Xihao Li, et al. NatGen 2020.
STAAR pipeline methods paper, Zilin Li, et al. NatMethods 2022.
STAAR pipeline github
Controlling SKAT function: Here’s a summary of the SKAT package functions - which are easier to understand than reading the notation in the SKAT papers. If you read the code you see each new implement is added sequentially and how weights work. Although, the ACAT git repo is independent.
ACAT git repo: https://github.com/yaowuliu/ACAT/blob/master/R/ACAT.R

Step-by-step explanation of ACAT.R

This discussion refers to code in the main ACAT function found at https://github.com/yaowuliu/ACAT.

Figure 1. Summary of the Proposed Methods ACAT, ACAT-V, and ACAT-O and the Relationship Among Them. From the ACAT paper

The R code defines several functions to perform the Aggregated Cauchy Association Test (ACAT) and the ACAT-V test.
- ACAT: This function combines p-values using the Cauchy distribution.
- ACAT_V: A set-based test that uses ACAT to combine the variant-level p-values.
- NULL_Model: Computes model parameters and residuals for ACAT-V.
- Get.marginal.pval: A helper function to calculate the marginal p-values for ACAT-V.
ACAT function
- a. It accepts Pvals (p-values), weights, and an optional is.check parameter to validate the input.
- b. Checks for NA, p-value range (0 to 1), and existence of both 0 and 1 p-values in the same column.
- c. If weights are not provided, equal weights are used. Otherwise, user-supplied weights are validated and standardized.
- d. The function calculates the Cauchy statistics and returns the ACAT p-value(s).
ACAT_V function
- a. It accepts G (genotype matrix), obj (output object of NULL_Model), weights.beta, weights, and mac.thresh.
- b. It checks for the validity of input weights.
- c. Based on the mac.thresh value, it decides to use the Burden test, the Cauchy method, or a combination of both.
- d. It calculates the final p-value and returns it.
NULL_Model function
- a. It accepts Y (outcome phenotypes) and Z (covariates).
- b. It determines if Y is continuous or binary.
- c. It fits a linear regression model if Y is continuous and a logistic model if Y is binary.
- d. It returns an object with model parameters and residuals.
Get.marginal.pval function
- a. It accepts G (genotype matrix) and obj (output object of NULL_Model).
- b. It checks the validity of the obj input.
- c. It calculates the marginal p-values and returns them.

The Aggregated Cauchy Association Test (ACAT) is a powerful and computationally efficient method designed to improve the analysis of rare and low-frequency genetic variants in sequencing studies. Traditional set-based tests can experience power loss when only a small proportion of variants are causal, and their power can be sensitive to factors such as the number, effect sizes, and effect directions of causal variants, as well as weight choices.

ACAT addresses these issues by combining variant-level p-values to create a set-based test called ACAT-V. ACAT-V is particularly powerful when there are only a few causal variants in a set, making it a valuable tool for genetic analysis. Additionally, ACAT can be used to create an omnibus test called ACAT-O by combining different variant-set-level p-values. ACAT-O incorporates the strengths of multiple complementary set-based tests, such as the burden test, sequence kernel association test (SKAT), and ACAT-V.

By analyzing extensive simulated data and real-world data from the Atherosclerosis Risk in Communities (ARIC) study, it has been demonstrated that ACAT-V complements other tests like SKAT and the burden test. Furthermore, ACAT-O consistently delivers more robust and higher power than alternative tests, making it a valuable addition to the toolkit of researchers working with sequencing studies.

ACAT is designed to combine p-values from multiple variants or tests rather than combining annotation scores directly. If you have p-values associated with each of the 5 annotation columns (CADD_score, MAF, GnomAD_AF, REVEL_score, ClinVar_score) for a single variant, you could potentially use ACAT to combine these p-values to obtain a single combined p-value for that variant. However, it’s essential to ensure that the p-values are valid and independent for ACAT to be effective.

To do this see the STAAR framework for this.

Figure 2. Slide from presentation of ACAT method.

Applying STAAR-O for multiple annotation weights

In a separate page, I discuss the STAAR method. The following passages are included in both pages since they related.

In the STAAR Nature Methods paper, the section Gene-centric analysis of the noncoding genome shows how the STAAR method can indeed be used to capitalize on the ACAT method to obtain a combined p-value from a set of annotations for a single variant. The STAAR framework incorporates multiple functional annotation scores into the RVATs (rare-variant association tests) to increase the power of association analysis. In this context, it uses the STAAR-O test, an omnibus test that aggregates annotation-weighted burden test, SKAT, and ACAT-V within the STAAR framework.

By incorporating multiple functional annotation scores, such as CADD, LINSIGHT, FATHMM-XF, and annotation principal components (aPCs), the STAAR method enhances the ability to detect associations between variants and traits of interest. Therefore, the STAAR framework can be used to leverage the strengths of the ACAT method and obtain a combined p-value from a set of annotations for a single variant or a set of variants.

Non-gene-centric analysis using dynamic windows with SCANG-STAAR

The SCANG-STAAR method is an improvement over the fixed-size sliding window RVAT in the STAAR framework. It proposes a dynamic window-based approach called SCANG-STAAR, which extends the SCANG procedure by incorporating multidimensional functional annotations. This method allows for flexible detection of locations and sizes of signal windows across the genome, as the locations of regions associated with a disease or trait are often unknown in advance, and their sizes may vary across the genome. Using a prespecified fixed-size sliding window for RVAT can lead to power loss if the prespecified window sizes do not align with the true locations of the signals.

The SCANG-STAAR method has two main procedures: SCANG-STAAR-S and SCANG-STAAR-B. SCANG-STAAR-S extends the SCANG-SKAT (SCANG-S) procedure by calculating the STAAR-SKAT (STAAR-S) p-value in each overlapping window by incorporating multiple variant functional annotations, instead of using just the MAF-weight-based SKAT p-value. SCANG-STAAR-B is based on the STAAR-Burden p-value. SCANG-STAAR-S has two advantages over SCANG-STAAR-B in detecting noncoding associations using dynamic windows: first, the effects of causal variants in a neighborhood in the noncoding genome tend to be in different directions, especially in intergenic regions; second, due to the different correlation structures of the two test statistics for overlapping windows, the genome-wide significance threshold of SCANG-STAAR-B is lower than that of SCANG-STAAR-S.

SCANG-STAAR also provides the SCANG-STAAR-O procedure, based on an omnibus p-value of SCANG-STAAR-S and SCANG-STAAR-B calculated by the ACAT method. However, unlike STAAR-O, the ACAT-V test is not incorporated into the omnibus test because it is designed for sparse alternatives, and as a result, it tends to detect the region with the smallest size that contains the most significant variant in the dynamic window procedure.

Figure 3. Slide from presentation of ACAT application in STAAR.

Multi-weight annotation analysis

The STAAR framework can be used to combine the p-values associated with each of the 5 annotation columns (CADD_score, MAF, GnomAD_AF, REVEL_score, ClinVar_score) for a single variant. STAAR incorporates multiple functional annotation scores as weights when constructing its statistics, making it suitable for combining p-values from different annotation columns to obtain a single combined p-value for that variant.

Main equations for ACAT

ACAT test statistic:

\[T_{ACAT} = \sum_{i=1}^{k} w_i \tan{[(0.5 - p_i)\pi]}\]

where $p_i$ are the p-values, and $w_i$ are non-negative weights.

P-value calculation for the ACAT test statistic:

\[p \text{-value} \approx 1 - \frac{1}{2} + \frac{\arctan{(T_{ACAT} / w)}}{\pi}\]

where $w = \sum_{i=1}^{k} w_i$.

ACAT is a general and flexible method of combining p-values, which can represent the statistical significance of different kinds of genetic variations in sequencing studies.
ACAT only aggregates p-values, so one can automatically control cryptic relatedness and/or population stratification by fitting appropriate models from which p-values are calculated through methods such as principal-component analysis or mixed models.
The null distribution of the test statistic $T_{ACAT}$ can be well approximated by a Cauchy distribution without the need for estimating and accounting for the correlation among p-values.
Calculating the p-value of ACAT requires almost negligible computation and is extremely fast.
The approximation is particularly accurate when ACAT has a very small p-value, which is useful in sequencing studies because only very small p-values can pass the stringent genome-wide significance threshold and are of particular interest.

tan and $\pi$

In the ACAT method, the “tan” and “π” functions are used to transform the p-values in such a way that they follow a standard Cauchy distribution under the null hypothesis. This transformation is essential to the ACAT method because it allows for an efficient and accurate combination of p-values, even when they are correlated.

The reason for using the tangent function (“tan”) specifically is because of its connection to the Cauchy distribution. The Cauchy distribution has some unique properties, such as having a heavy tail, which make it suitable for handling correlated p-values in this context. The transformation function used in the ACAT method, given by $tan((0.5 - p_i) \pi)$, ensures that if the p-value $p_i$ is from the null distribution, the transformed value will follow a standard Cauchy distribution.

The constant $\pi$ (Pi) is used in the formula because it is a natural component of the tangent function. In the context of the ACAT method, $\pi$ is used to scale the input of the tangent function, which is necessary to map the range of p-values (0 to 1) to the entire domain of the tangent function. This ensures that the transformed values will follow the desired Cauchy distribution.

Therefore, the “tan” and $\pi$ functions in the ACAT method are used to transform p-values so that they follow a standard Cauchy distribution under the null hypothesis, which allows for an efficient and accurate combination of correlated p-values.

Original R code from yaowuliu

This code is the main ACAT function found at https://github.com/yaowuliu/ACAT

#'
#' Aggregated Cauchy Assocaition Test
#'
#' A p-value combination method using the Cauchy distribution.
#'
#'
#'
#' @param weights a numeric vector/matrix of non-negative weights for the combined p-values. When it is NULL, the equal weights are used.
#' @param Pvals a numeric vector/matrix of p-values. When it is a matrix, each column of p-values is combined by ACAT.
#' @param is.check logical. Should the validity of \emph{Pvals} (and \emph{weights}) be checked? When the size of \emph{Pvals} is large and one knows \emph{Pvals} is valid, then the checking part can be skipped to save memory.
#' @return The p-value(s) of ACAT.
#' @author Yaowu Liu
#' @examples p.values<-c(2e-02,4e-04,0.2,0.1,0.8);ACAT(Pvals=p.values)
#' @examples ACAT(matrix(runif(1000),ncol=10))
#' @references Liu, Y., & Xie, J. (2019). Cauchy combination test: a powerful test with analytic p-value calculation
#' under arbitrary dependency structures. \emph{Journal of American Statistical Association},115(529), 393-402. (\href{https://amstat.tandfonline.com/doi/abs/10.1080/01621459.2018.1554485}{pub})
#' @export
ACAT<-function(Pvals,weights=NULL,is.check=TRUE){
    Pvals<-as.matrix(Pvals)
    if (is.check){
        #### check if there is NA
        if (sum(is.na(Pvals))>0){
            stop("Cannot have NAs in the p-values!")
        }
        #### check if Pvals are between 0 and 1
        if ((sum(Pvals<0)+sum(Pvals>1))>0){
            stop("P-values must be between 0 and 1!")
        }
        #### check if there are pvals that are either exactly 0 or 1.
        is.zero<-(colSums(Pvals==0)>=1)
        is.one<-(colSums(Pvals==1)>=1)
        if (sum((is.zero+is.one)==2)>0){
            stop("Cannot have both 0 and 1 p-values in the same column!")
        }

        if (sum(is.zero)>0){
            warning("There are p-values that are exactly 0!")
        }
        if (sum(is.one)>0){
            warning("There are p-values that are exactly 1!")
        }

    }
    #### Default: equal weights. If not, check the validity of the user supplied weights and standadize them.
    if (is.null(weights)){
        is.weights.null<-TRUE
    }else{
        is.weights.null<-FALSE
        weights<-as.matrix(weights)
        if (sum(dim(weights)!=dim(Pvals))>0){
            stop("The dimensions of weights and Pvals must be the same!")
        }else if (is.check & (sum(weights<0)>0)){
            stop("All the weights must be nonnegative!")
        }else{
            w.sum<-colSums(weights)
            if (sum(w.sum<=0)>0){
                stop("At least one weight should be positive in each column!")
            }else{
                for (j in 1:ncol(weights)){
                    weights[,j]<-weights[,j]/w.sum[j]
                }
            }
        }

    }

    #### check if there are very small non-zero p values and calcuate the cauchy statistics
    is.small<-(Pvals<1e-15)
    if (is.weights.null){
         Pvals[!is.small]<-tan((0.5-Pvals[!is.small])*pi)
         Pvals[is.small]<-1/Pvals[is.small]/pi
         cct.stat<-colMeans(Pvals)
    }else{
         Pvals[!is.small]<-weights[!is.small]*tan((0.5-Pvals[!is.small])*pi)
         Pvals[is.small]<-(weights[is.small]/Pvals[is.small])/pi
         cct.stat<-colSums(Pvals)
    }
    #### return the ACAT p value(s).
    pval<-pcauchy(cct.stat,lower.tail = F)
    return(pval)
}

#'
#' A set-based test that uses ACAT to combine the variant-level p-values.
#'
#'
#' @param G a numeric matrix or dgCMatrix with each row as a different individual and each column as a separate gene/snp. Each genotype should be coded as 0, 1, 2.
#' @param obj an output object of the \code{\link{NULL_Model}} function.
#' @param weights.beta a numeric vector of parameters for the beta weights for the weighted kernels. If you want to use your own weights, please use the “weights” parameter. It will be ignored if “weights” parameter is not null.
#' @param weights a numeric vector of weights for the SNP p-values. When it is NULL, the beta weight with the “weights.beta” parameter is used.
#' @param mac.thresh a threshold of the minor allele count (MAC). The Burden test will be used to aggregate the SNPs with MAC less than this thrshold.
#' @return The p-value of ACAT-V.
#' @details The Burden test is first used to aggregate very rare variants with Minor Allele Count (MAC) < \emph{mac.thresh} (e.g., 10), and a Burden p-value is obtained. For each of the variants with MAC >= \emph{mac.thresh}, a variant-level p-value is calculated. Then, ACAT is used to combine the variant-level p-values and the Burden test p-value of very rare variants.
#'
#' If \emph{weights.beta} is used, then the weight for the Burden test p-value is demetermined by the average Minor Allele Frequency (MAF) of the variants with MAC < \emph{mac.thresh}; if the user-specified \emph{weights} is used, then the weight for the Burden test p-value is the average of \emph{weights} of the variants with MAC < \emph{mac.thresh}.
#'
#' Note that the \emph{weights} here are for the SNP p-vlaues. In SKAT, the weights are for the SNP score test statistics. To transfrom the SKAT weights to the \emph{weights} here, one can use the formula that \emph{weights} = (skat_weights)^2*MAF*(1-MAF).
#'
#' @author Yaowu Liu
#' @references Liu, Y., et al. (2019). ACAT: A fast and powerful p value combination
#' method for rare-variant analysis in sequencing studies.
#' \emph{American Journal of Humann Genetics 104}(3), 410-421.
#' (\href{https://www.sciencedirect.com/science/article/pii/S0002929719300023}{pub})
#' @examples  library(Matrix)
#' @examples  data(Geno)
#' @examples  G<-Geno[,1:100] # Geno is a dgCMatrix of genotypes
#' @examples  Y<-rnorm(nrow(G)); Z<-matrix(rnorm(nrow(G)*4),ncol=4)
#' @examples  obj<-NULL_Model(Y,Z)
#' @examples  ACAT_V(G,obj)
#' @export
ACAT_V<-function(G,obj,weights.beta=c(1,25),weights=NULL,mac.thresh=10){
    ### check weights
    if (!is.null(weights) && length(weights)!=ncol(G)){
        stop("The length of weights must equal to the number of variants!")
    }

    mac<-Matrix::colSums(G)
    ### remove SNPs with mac=0
    if (sum(mac==0)>0){
        G<-G[,mac>0,drop=FALSE]
        weights<-weights[mac>0]
        mac<-mac[mac>0]
        if (length(mac)==0){
            stop("The genotype matrix do not have non-zero element!")
        }
    }
    ### p and n
    p<-length(mac)
    n<-nrow(G)
    ###

    if (sum(mac>mac.thresh)==0){  ## only Burden
        pval<-Burden(G,obj, weights.beta = weights.beta, weights = weights)
    }else if (sum(mac<=mac.thresh)==0){ ## only cauchy method
        if (is.null(weights)){
            MAF<-mac/(2*n)
            W<-(dbeta(MAF,weights.beta[1],weights.beta[2])/dbeta(MAF,0.5,0.5))^2
        }else{
            W<-weights
        }

        Mpvals<-Get.marginal.pval(G,obj)
        pval<-ACAT(Mpvals,W)
    }else{  ## Burden + Cauchy method
        is.very.rare<-mac<=mac.thresh
        weights.sparse<-weights[is.very.rare]
        weights.dense<-weights[!is.very.rare]
        pval.dense<-Burden(G[,is.very.rare,drop=FALSE],obj, weights.beta = weights.beta, weights = weights.sparse)

        Mpvals<-Get.marginal.pval(G[,!is.very.rare,drop=FALSE],obj)

        Mpvals<-c(Mpvals,pval.dense)
        if (is.null(weights)){
            MAF<-mac/(2*n)
            mafs<-c(MAF[!is.very.rare],mean(MAF[is.very.rare])) ## maf for p-values
            W<-(dbeta(mafs,weights.beta[1],weights.beta[2])/dbeta(mafs,0.5,0.5))^2
        }else{
            W<-c(weights.dense,mean(weights.sparse))
        }


        is.keep<-rep(T,length(Mpvals))
        is.keep[which(Mpvals==1)]<-F  ## remove p-values of 1.
        pval<-ACAT(Mpvals[is.keep],W[is.keep])
    }
    return(pval)
}

#'
#'
#' Get parameters and residuals from the NULL model
#'
#' Compute model parameters and residuals for ACAT-V
#'
#'
#' @param Y a numeric vector of outcome phenotypes.
#' @param Z a numeric matrix of covariates. Z must be full-rank. Do not include intercept in Z. The intercept will be added automatically.
#' @return This function returns an object that has model parameters and residuals of the NULL model of no association between genetic variables and outcome phenotypes. After obtaining it, please use \code{\link{ACAT_V}} function to conduct the association test.
#' @details \emph{Y} could only be continuous or binary. If \emph{Y} is continuous, a linear regression model is fitted. If \emph{Y} is binary, it must be coded as 0,1 and a logistic model is fitted.
#' @author Yaowu Liu
#' @examples  Y<-rnorm(10000)
#' @examples  Z<-matrix(rnorm(10000*4),ncol=4)
#' @examples  obj<-NULL_Model(Y,Z)
#' @export
NULL_Model<-function(Y,Z=NULL){
    n<-length(Y)
    #### check the type of Y
    if ((sum(Y==0)+sum(Y==1))==n){
        out_type<-"D"
    }else{
        out_type<-"C"
    }
    #### Add intercept
    Z.tilde<-cbind(rep(1,length(Y)),Z)
    if (out_type=="C"){
        #### estimate of sigma square
        Z.med<-Z.tilde%*%solve(chol(t(Z.tilde)%*%Z.tilde))   ## Z.med%*%t(Z.med) is the projection matrix of Z.tilde
        Y.res<-as.vector(Y-(Y%*%Z.med)%*%t(Z.med))
        sigma2<-sum(Y.res^2)/(n-ncol(Z.med))
        #### output
        res<-list()
        res[["out_type"]]<-out_type
        res[["Z.med"]]<-Z.med
        res[["Y.res"]]<-Y.res
        res[["sigma2"]]<-sigma2
    }else if (out_type=="D"){
        #### fit null model
        g<-glm(Y~0+Z.tilde,family = "binomial")
        prob.est<-g[["fitted.values"]]
        #### unstandarized residuals
        Y.res<-(Y-prob.est)
        ### Sigma when rho=0
        sigma2.Y<-prob.est*(1-prob.est)  ### variance of each Y_i
        ### output
        res<-list()
        res[["out_type"]]<-out_type
        res[["Z.tilde"]]<-Z.tilde
        res[["Y.res"]]<-Y.res
        res[["sigma2.Y"]]<-sigma2.Y
    }
    return(res)
}




Get.marginal.pval<-function(G,obj){
    ### check obj
    if (names(obj)[1]!="out_type"){
        stop("obj is not calculated from MOAT_NULL_MODEL!")
    }else{
        out_type<-obj[["out_type"]]
        if (out_type=="C"){
            if (!all.equal(names(obj)[2:length(obj)],c("Z.med","Y.res","sigma2"))){
                stop("obj is not calculated from MOAT_NULL_MODEL!")
            }else{
                Z.med<-obj[["Z.med"]]
                Y.res<-obj[["Y.res"]]
                n<-length(Y.res)
                SST<-obj[["sigma2"]]*(n-ncol(Z.med))
            }
        }else if (out_type=="D"){
            if (!all.equal(names(obj)[2:length(obj)],c("Z.tilde","Y.res","sigma2.Y"))){
                stop("obj is not calculated from MOAT_NULL_MODEL!")
            }else{
                Z.tilde<-obj[["Z.tilde"]]
                Y.res<-obj[["Y.res"]]
                sigma2.Y<-obj[["sigma2.Y"]]
                n<-length(Y.res)
            }
        }
    }

    if (class(G)!="matrix" && class(G)!="dgCMatrix"){
        stop("The class of G must be matrix or dgCMatrix!")
    }

    if (out_type=="C"){
        G_tX.med<-as.matrix(Matrix::crossprod(Z.med,G))
        ### Sigma^2 of G
        Sigma2.G<-Matrix::colSums(G^2)-Matrix::colSums(G_tX.med^2)
        SSR<-as.vector((Y.res%*%G)^2/Sigma2.G)
        SSR[Sigma2.G<=0]<-0
        df.2<-n-1-ncol(Z.med)
        t.stat<-suppressWarnings(sqrt(SSR/((SST-SSR)/df.2)))
        marginal.pvals<-2*pt(t.stat,(n-1-ncol(Z.med)),lower.tail = FALSE)
    }else if (out_type=="D"){
        Z.stat0<-as.vector(Y.res%*%G)
        ### Sigma when rho=0
        tG_X.tilde_sigma2<-as.matrix(Matrix::crossprod(G,Z.tilde*sigma2.Y))
        Sigma2.G<-Matrix::colSums(G^2*sigma2.Y)-diag(tG_X.tilde_sigma2%*%solve(t(Z.tilde)%*%(Z.tilde*sigma2.Y))%*%t(tG_X.tilde_sigma2))
        marginal.pvals<-2*pnorm(abs(Z.stat0)/sqrt(Sigma2.G),lower.tail = FALSE)
    }

    return(marginal.pvals)
}


Burden<-function(G,obj,kernel="linear.weighted",weights.beta=c(1,25),weights=NULL){
    ### check obj
    if (names(obj)[1]!="out_type"){
        stop("obj is not calculated from NULL_MODEL!")
    }else{
        out_type<-obj[["out_type"]]
        if (out_type=="C"){
            if (!all.equal(names(obj)[2:length(obj)],c("Z.med","Y.res","sigma2"))){
                stop("obj is not calculated from NULL_MODEL!")
            }else{
                Z.med<-obj[["Z.med"]]
                Y.res<-obj[["Y.res"]]/sqrt(obj[["sigma2"]])  ## rescaled residules
                n<-length(Y.res)
            }
        }else if (out_type=="D"){
            if (!all.equal(names(obj)[2:length(obj)],c("Z.tilde","Y.res","sigma2.Y"))){
                stop("obj is not calculated from NULL_MODEL!")
            }else{
                Z.tilde<-obj[["Z.tilde"]]
                Y.res<-obj[["Y.res"]]
                sigma2.Y<-obj[["sigma2.Y"]]
                n<-length(Y.res)
            }
        }
    }
    ### MAF
    MAF<-Matrix::colSums(G)/(2*dim(G)[1])
    p<-length(MAF)
    #### weights
    if (kernel=="linear.weighted"){
        if (is.null(weights)){
            W<-dbeta(MAF,weights.beta[1],weights.beta[2])
        }else{
            if (length(weights)==p){
                W<-weights
            }else{
                stop("The length of weights must equal to the number of variants!")
            }
        }

    }else if (kernel=="linear"){
        W<-rep(1,p)
    }else{
        stop("The kernel name is not valid!")
    }

    ###### if G is sparse or not
    if (class(G)=="matrix" || class(G)=="dgCMatrix"){
        if (out_type=="C"){
            Z.stat.sum<-as.vector((Y.res%*%G)%*%W)
            Gw<-G%*%W
            sigma.z<-sqrt(sum(Gw^2)-sum((t(Z.med)%*%(Gw))^2))
        }else if (out_type=="D"){
            Z.stat.sum<-as.vector((Y.res%*%G)%*%W)
            Gw<-as.vector(G%*%W)
            sigma.z<-sum(Gw^2*sigma2.Y)-((Gw*sigma2.Y)%*%Z.tilde)%*%solve(t(Z.tilde)%*%(Z.tilde*sigma2.Y))%*%t((Gw*sigma2.Y)%*%Z.tilde)
            sigma.z<-as.vector(sqrt(sigma.z))
        }
    }else{
        stop("The class of G must be matrix or dgCMatrix!")
    }

    V<-Z.stat.sum/sigma.z
    Q<-V^2   ## Q test statistic
    pval<-1-pchisq(Q,df=1)
    return(pval)
}

Analysis of methods

Sat, 16 Sep 2023 00:00:00 +0200

Analysis of methods

The fianl comparison results
- Scores per sample and Bland-Altman plot
- Correlation test result and repeatability coefficients
Code
Summary of the analysis of method comparison studies

Altman, D. G., and J. M. Bland. “Measurement in Medicine: The Analysis of Method Comparison Studies.” Journal of the Royal Statistical Society. Series D (The Statistician), vol. 32, no. 3, 1983, pp. 307–17. JSTOR, https://doi.org/10.2307/2987937.

The paper is a pivotal guide discussing the analysis of method comparison studies, particularly in the field of medicine. It proposes a pragmatic approach to analyze such studies, stressing the importance of simplicity especially when the results need to be explained to non-statisticians. I work through it here to compare if a new methods is as good or better than an existing one for clinical sepsis scores with example data.

For more similar papers see the series of BMJ statistical notes by Altman & Bland ( lit-altman_bland.md ).

The fianl comparison results

Scores per sample and Bland-Altman plot

Figure 1. Scores per sample and Bland-Altman plot as produced by the provided R code.

Correlation test result and repeatability coefficients

Results of the analysis as produced by the provided R code:

$Repeatability_Score1
[1] 4.674812

$Repeatability_Score2
[1] 4.267645

$Correlation_Test

	Pearson's product-moment correlation

data:  df$score1 and df$score2
t = 21.693, df = 197, p-value < 2.2e-16
alternative hypothesis: true correlation is not equal to 0
95 percent confidence interval:
 0.7931229 0.8763442
sample estimates:
      cor 
0.8395928 

Repeatability Scores:

Repeatability Score 1: 4.674812
Repeatability Score 2: 4.267645

The repeatability scores are measures of how consistent each method is. In this case, both methods seem to have fairly similar repeatability scores, indicating a similar level of consistency or reliability within each method. Without context or a benchmark to compare to, it’s challenging to definitively say whether these scores are good or not, but the similarity suggests comparable repeatability.

Correlation Test:

Pearson’s Correlation Coefficient (r): 0.8395928
95% Confidence Interval for r: [0.7931229, 0.8763442]
P-value: $< 2.2e-16$

The Pearson’s correlation coefficient is quite high, indicating a strong positive linear relationship between the scores from the two methods. The nearly zero p-value (less than 2.2e-16) strongly suggests that this correlation is statistically significant, and it’s highly unlikely that this observed correlation occurred by chance.

Conclusion:

Considering the high correlation coefficient and the comparable repeatability scores, it seems that the new method is quite similar to the old one in terms of both reliability (as indicated by the repeatability scores) and agreement (as indicated by the correlation coefficient).

Code

# Here is a really cool set of notes in BMJ about all kinds of clinical data analysis.
# https://www-users.york.ac.uk/~mb55/pubs/pbstnote.htm

# For example, if you every like to go further with the score that you working on, this paper is very famous for showing how to compare two such clinical score methods to show if the new one performs better. 

# https://sci-hub.hkvisa.net/10.2307/2987937
# (Measurement in Medicine: The Analysis of Method Comparison Studies Author(s): D. G. Altman and J. M. Bland 1983)

# Example ----
# psofa.score - Total number of organ failures according to 2017 pSOFA definitions (Matics et al. (2017) (PMID 28783810)). The classification of organ failures is based on the worst vital signs and the worst lab values during the first 7 days from blood culture sampling.

# pelod.score - Total number of organ failures according to PELOD-2 definitions (Leteurtre et al. (2013) (PMID 23685639)). The classification of organ failures is based on the worst vital signs and the worst lab values during the first 7 days from blood culture sampling.

library(ggplot2)
library(dplyr)
library(tidyr)
library(patchwork)

# Read data
df <- read.csv(file = "../data/example_sepsis_scores.tsv", sep = " ")

# Take 200 rows as an example
df <- df[1:200, ]

# Renaming columns and adding index column
df$score1 <- df$psofa.score
df$score2 <- df$pelod.score
df$index <- rownames(df)

# Adding a small amount of random noise to each value
# NOTE: THIS WOULD BE USED FOR ADDING SOME NOISE TO MAKE THE DATA MORE ANONYMOUS FOR PUBLISHING AN EXAMPLE FIGURE
set.seed(123) # Setting a seed to ensure reproducibility
df$score1 <- df$score1 + rnorm(nrow(df), mean = 0, sd = 1)
df$score2 <- df$score2 + rnorm(nrow(df), mean = 0, sd = 1)

# Calculate necessary statistics: the average and the difference of score1 and score2.
df$means <- (df$score1 + df$score2) / 2
df$diffs <- df$score1 - df$score2

# Compute repeatability coefficients for each method separately
repeatability_score1 <- sd(df$score1, na.rm = TRUE)
repeatability_score2 <- sd(df$score2, na.rm = TRUE)

# Compute correlation to check independence of repeatability from the size of the measurements
cor_test <- cor.test(df$score1, df$score2, method = "pearson")

# Average difference (aka the bias)
bias <- mean(df$diffs, na.rm = TRUE) 

# Sample standard deviation
sd <- sd(df$diffs, na.rm = TRUE) 

# Limits of agreement
upper_loa <- bias + 1.96 * sd
lower_loa <- bias - 1.96 * sd

# Additional statistics for confidence intervals
n <- nrow(df)
conf_int <- 0.95
t1 <- qt((1 - conf_int) / 2, df = n - 1)
t2 <- qt((conf_int + 1) / 2, df = n - 1)
var <- sd^2
se_bias <- sqrt(var / n)
se_loas <- sqrt(3 * var / n)
upper_loa_ci_lower <- upper_loa + t1 * se_loas
upper_loa_ci_upper <- upper_loa + t2 * se_loas
bias_ci_lower <- bias + t1 * se_bias
bias_ci_upper <- bias + t2 * se_bias
lower_loa_ci_lower <- lower_loa + t1 * se_loas
lower_loa_ci_upper <- lower_loa + t2 * se_loas

# Create Bland-Altman plot
p1 <- df |>
  ggplot(aes(x = means, y = diffs)) + 
  geom_point(color = "red", alpha = .5) +
  ggtitle("Bland-Altman plot for PELOD and pSOFA scores") + 
  ylab("Difference between\ntwo scores") + 
  xlab("Average of two scores") +
  theme_bw() +
  geom_hline(yintercept = c(lower_loa, bias, upper_loa), linetype = "solid") +
  geom_hline(yintercept = c(lower_loa_ci_lower, lower_loa_ci_upper, upper_loa_ci_lower, upper_loa_ci_upper), linetype="dotted") +
  geom_hline(yintercept = se_bias)

# Create scatter plot with a line of equality
p2 <- df |>
  ggplot(aes(x=score1, y=score2)) +
  geom_point(color = "red", alpha = .5) +
  geom_abline(intercept = 0, slope = 1) +
  theme_bw() + 
  ggtitle("PELOD and pSOFA scores per sample") + 
  ylab("pelod.score") + 
  xlab("psofa.score") 

# Combine plots
patch1 <- (p2 / p1) + plot_annotation(tag_levels = 'A')
patch1

# Output the correlation test result and repeatability coefficients
list(
  Repeatability_Score1 = repeatability_score1,
  Repeatability_Score2 = repeatability_score2,
  Correlation_Test = cor_test
)

Summary of the analysis of method comparison studies

In their landmark paper, D.G. Altman and J.M. Bland outline a structured approach to evaluating whether a new method of medical measurement is as good as or better than an existing one. The approach encapsulates several critical components, emphasizing not only statistical analyses but also the importance of effective communication, especially to non-expert audiences. Key points from the paper:

Bland-Altman Plot
- Introduced as a graphical method to assess the agreement between two different measurement techniques. This method involves plotting the difference between two methods against their mean, which assists in identifying any biases and analyzing the limits of agreement between the two methods.
Bias and Limits of Agreement
- The authors recommend calculating the bias (the mean difference between two methods) and limits of agreement (bias ± 1.96 times the standard deviation of the differences) to quantify the agreement between the two methods. A smaller bias and narrower limits of agreement generally indicate that a new method might be comparable or superior to the existing one.
Investigating Relationship with Measurement Magnitude
- Encourages the investigation of whether the differences between the methods relate to the measurement’s magnitude. Transformations or regression approaches might be necessary depending on the observed association, to correct for it.
Repeatability
- Assessment: It’s crucial to assess repeatability for each method separately using replicated measurements on a sample of subjects. This analysis derives from the within-subject standard deviation of the replicates.
- Graphical Methods and Correlation Tests: Apart from Bland-Altman plots, graphical methods (like plotting standard deviation against the mean) and correlation coefficient tests are suggested for examining the independence of repeatability from the size of measurements.
- Potential Influences: It highlights the possible influences on measurements, such as observer variability, time of day, or the position of the subject, and differentiates between repeatability and reproducibility (agreement of results under different conditions).
Comparison of Methods
- The core emphasis is on directly comparing results obtained by different methods to determine if one can replace another without compromising accuracy for the intended purpose of the measurement. Initial data plotting is encouraged, ideally plotting the difference between methods against the average of the methods, providing insight into disagreement, outliers, and potential trends. Testing for independence between method differences and the size of measurements is necessary, as it can influence the analysis and interpretation of bias and error.
Addressing Alternative Analyses
- The paper discusses alternative approaches like least squares regression, principal component analysis, and regression models with errors in both variables, but finds them to generally add complexity without necessarily improving the simple comparison intended.
Effective Communication
- The authors emphasize the importance of communicating results effectively to non-experts, such as clinicians, to facilitate practical application of the findings.
Challenges in Method Comparison Studies
- The paper highlights the challenges faced in method comparison studies, primarily due to the lack of professional statistical expertise and reliance on incorrect methods replicated from existing literature. It calls for improved awareness among statisticians about this issue and encourages journals to foster the use of appropriate techniques through peer review.

Thus, we can perform an objective evaluation of whether a new measurement method is as good or potentially better than an existing one by assessing agreement, bias, and repeatability, among other factors.

Correlation, regression and repeated data

Tue, 29 Jun 2021 00:00:00 +0200

Correlation, regression and repeated data

Introduction
Correlation within subjects, part 1
Correlation within subjects, part 2
References

Introduction

This topic is introduced as the first paper (Bland & Altman, 1994) in a series of BMJ statistical notes by Altman & Bland ( lit-altman_bland.md ): 1. Bland JM, Altman DG. (1994) Correlation, regression and repeated data. 308, 896. ¹

It concerns the analysis of paired data where there is more than one observation per subject. They point out that it could be highly misleading to analyse such data by combining repeated observations from several subjects and then calculating the correlation coefficient as if the data were a simple sample.

Many researchers would assume that it is acceptable to gather repeated measurements for individuals and put all the data together.

They use simulated data showing five pairs of measurements of two uncorrelated variables X and Y for subjects 1, 2, 3, 4, and 5. Using each subject’s mean values, they show correlation coefficient r=-0.67, df=3, P=0.22. However, when they put all 25 observations together they get r=-0.47, df=23, P=0.02. When the calculation is performed as if they have 25 subjects, the number of degrees of freedom for the significance test is increased incorrectly and a spurious significant difference is produced. Thus demonstrating that one should not mix observations from different subjects indiscriminately, whether using correlation or the closely related regression analysis.

Correlation within subjects, part 1

The methods to use in these circumstances are later discussed in another note for the BMJ series (Bland & Altman, 1995), number 11 on the list lit-altman_bland.md : 11. Bland JM, Altman DG. (1995) Calculating correlation coefficients with repeated observations: Part 1, correlation within subjects. 310, 446.

Notes: I am replacing their terms for my notes:

X = Paco2
Y = pHi

In this note they show an example table using 8 subjects with 4-8 observations for X and Y (table I):

Subject	Y	X
1	6.68	3.97
1	6.53	4.12
…	…	…

If subject’s average Y is related to the subject’s average X
- We can use the correlation between the subject means, which they shall describe in a subsequent note.
If an increase in Y within the individual was associated with an increase in X
- We want to remove the differences between subjects and look only at changes within.

To look at variation within the subject we can use multiple regression.

Make one variable, X or Y, the outcome variable and the other variable and the subject the predictor variables.
The subject is treated as a categorical factor using dummy variables and so has seven degrees of freedom.
Using an analysis of variance table for the regression (table II) shows how the variability in Y can be partitioned into components due to different sources.
- Also known as analysis of covariance
- Equivalent to fitting parallel lines through each subject’s data (Figure 1.)

Source of variation	Degrees of freedom	Sum of squares	Mean square	Variance ratio (F)	Probability
Subjects	7	2.9661	0.4237	48.3	$<$0.0001
X	1	0.1153	0.1153	13.1	0.0008
Residual	38	0.3337	0.0088
Total	46	3.3139	0.0720

Table II. Analysis of variance for the data in table I (as shown in Altman & Bland). At the end of this page, this table is reproduced based on the original data and new R code, as shown. (Note that there is a slight varition between the published version and my replicated version of table II and Figure 1 below; probably due to a minor data entry error by the publisher or authors).

The residual sum of squares in table II represents the variation about regression lines.
This removes the variation due to subjects (and any other nuisance variables which might be present) and express the variation in Y due to X as a proportion of what’s left:
- (Sum of squares for X)/(Sum of squares for X + residual sum of squares).
The magnitude of the correlation coefficient within subjects is the square root of this proportion.
- For table II this is: $\sqrt{ \frac{0.1153}{0.1153+0.3337} } = 0.51$
- The sign of the correlation coefficient is given by the sign of the regression coefficient for X.
Regression slope is -0.108
So the correlation coefficient within subjects is -0.51.
The P value is found either from:
- F test in the associated analysis of variance table,
- t test for the regression slope.
- It doesn’t matter which variable we regress on which;
- we get the same correlation coefficient and P value either way.

Incorrectly calculating the correlation coefficient by ignoring the fact that we have 47 observations on only 8 subjects, would produce -0.07, P=0.7.

Figure 1. Recreation of “(Y) pH against (X) PaCO2 for eight subjects, with parallel lines fitted for each subject” as used in (Bland & Altman, 1995). Interestingly, replotting this data shows that their figure was not fully accurate (forgivable before the days of Rstudio in 1995, and not important for this example).

Correlation within subjects, part 2

The second part shows how to find the correlation between the subject means (Bland & Altman, 1995), number 12 on the list lit-altman_bland.md : 12. Bland JM, Altman DG. (1995) Calculating correlation coefficients with repeated observations: Part 2, correlation between subjects. 310, 633.

In this note they show the example table using the same 8 subjects with one mean observation for X and Y:

Subject	Y	X	Number
1	6.49	4.04	4
2	7.05	5.37	4
3	7.36	4.83	9
4	7.33	5.31	5
5	7.31	4.40	8
6	7.32	4.92	6
7	6.91	6.60	3
8	7.12	4.78	8

If subject’s average Y is related to the subject’s average X
- We can use the correlation between the subject means.

They calculate the usual correlation coefficient for the mean Y and mean X; r=0.09, P=0.8. Does not take into account the different numbers of measurements on each subject.

Does this matter?:
- Depends on how different the numbers of observations are
- whether the measurements within subjects vary much compared with the means between subjects

We can calculate a weighted correlation coefficient using the number of observations as weights. Many computer programs will calculate this, but it is not difficult to do by hand.

They denote the mean Y and X for subject i by $\bar{x}_i$ and $\bar{y}_i$,
the number of observations for subject i by $m_i$,
and the number of subjects by $n$.
The weighted mean of the $\bar{x}_i$ is $\frac{ \sum{ m_i \bar{x}_i } }{ \sum{ m_i } }$

In the usual case, where there is one observation per subject, the $m_i$ are all one and this formula gives the usual mean $\frac{ \sum{\bar{x}_i} }{n}$.

An easy way to calculate the weighted correlation coefficient is to replace each individual observation by its subject mean. Thus the table would yield 47 pairs of observations, the first four of which would each be pH=6.49 and Paco2=4.04, and so on.

If we use the usual formula for the correlation coefficient on the expanded data we will get the weighted correlation coefficient. However, we must be careful when it comes to the P value. We have only 8 observations (n in general), not 47. We should ignore any P value printed by our computer program, and use a statistical table instead.

The formula for a weighted correlation coefficient is:

$$ \frac{ \sum{m_i \bar{x}_i \bar{y}_i} - \sum{m_i \bar{x}_i} \sum{m_i \bar{y}_i} \mathbin{/} \sum{m_i} }{ \sqrt{ ( \sum{m_i \bar{y}_i^2} - (\sum{m_i \bar{y}_i})^2 \mathbin{/} \sum{m_i} ) ( \sum{m_i \bar{y}_i^2} - (\sum{m_i \bar{y}_i})^2 \mathbin{/} \sum{m_i} )} } $$

where all summations are from $i=1$ to $n$. When all the $m_i$ are equal they cancel out, giving the usual formula for a correlation coefficient.

For the data in the table the weighted correlation coefficient is r=0.08, P=0.9. There is no evidence that subjects with a high Y also have a high X. However, as they have already shown in part 1, within the subject a rise in Y was associated with a fall in X.

## Code and raw data for Table I, Analysis of variance table II, and Figure 1
df <- data.frame (
  Subject = c("1", "1", "1", "1", "2", "2", "2", "2", "3", "3", "3", "3", "3", "3", "3", "3", "3", "4", "4", "4", "4", "4", "5", "5", "5", "5", "5", "5", "5", "5", "6", "6", "6", "6", "6", "6", "7", "7", "7", "8", "8", "8", "8", "8", "8", "8", "8"),

  Y = c(6.68, 6.53, 6.43, 6.33, 6.85, 7.06, 7.13, 7.17, 7.4, 7.42, 7.41, 7.37, 7.34, 7.35, 7.28, 7.3, 7.34, 7.36, 7.33, 7.29, 7.3, 7.35, 7.35, 7.3, 7.3, 7.37, 7.27, 7.28, 7.32, 7.32, 7.38, 7.3, 7.29, 7.33, 7.31, 7.33, 6.86, 6.94, 6.92, 7.19, 7.29, 7.21, 7.25, 7.2, 7.19, 6.77, 6.82),

  X = c(3.97, 4.12, 4.09, 3.97, 5.27, 5.37, 5.41, 5.44, 5.67, 3.64, 4.32, 4.73, 4.96, 5.04, 5.22, 4.82, 5.07, 5.67, 5.1, 5.53, 4.75, 5.51, 4.28, 4.44, 4.32, 3.23, 4.46, 4.72, 4.75, 4.99, 4.78, 4.73, 5.12, 4.93, 5.03, 4.93, 6.85, 6.44, 6.52, 5.28, 4.56, 4.34, 4.32, 4.41, 3.69, 6.09, 5.58)
)

# Run the Analysis of Variance with mutiple variable
name=aov(Y ~ Subject + X, data = df) #runs the ANOVA test
ls(name) #lists the items stored by the test.
summary(name) #give the basic ANOVA output.

# Output the column totals to match Altman & Bland table
df <- as.data.frame(unlist( summary(name) ))

sum(df[1:3,]) # Total Degrees of freedom
sum(df[4:6,]) # Total Sum Sq
sum(df[7:9,]) # Total Mean Sq

	Degrees of freedom	Sum of squares	Mean Square	Variance ratio (F)	Probability
df$Subject	7	2.8648	0.4093	46.60	< 2e-16
df$X	1	0.1153	0.1153	13.13	0.000847
Residuals	38	0.3337	0.0088

Replicated version of Table II. Analysis of variance for the data in table I. Default R output headings modified: Degrees of freedom (Df), Sum of squares (Sum Sq), Mean square (Mean Sq), Variance ratio F (F value), Probability (Pr(>F)). (Repeated note: there is a slight varition between the published version and my replicated version of table II and Figure 1; probably due to a minor data entry error by the publisher or authors).

# code used to produce Figure 1. 

require(ggplot2)
ggplot(df, aes(x = X, y = Y, group=Subject, color = Subject) ) +
  geom_point() +
  geom_smooth(aes(color = Subject), method = "lm", formula = y ~ x, , se = FALSE)

# The dataset is cited by Bland & Altman 1995 as: "Boyd O, Mackay CJ, Lamb G, Bland JM, Grounds RM, Bennett ED.Comparison of clinical information gained from routine blood-gas analysis and from gastric tonometry for intramural pH.Lancet1993;341:142–6."

References

Bland, J. M., & Altman, D. G. (1994). Correlation, regression, and repeated data. BMJ: British Medical Journal, 308(6933), 896.
Bland, J. M., & Altman, D. G. (1995). Statistics notes: Calculating correlation coefficients with repeated observations: Part 1—correlation within subjects. Bmj, 310(6977), 446.
Bland, J. M., & Altman, D. G. (1995). Calculating correlation coefficients with repeated observations: Part 2—Correlation between subjects. Bmj, 310(6980), 633.

Footnote 1 This article is almost identical to the original version in acknowledgment to Altman and Bland. It is adapted here as part of a set of curated, consistent, and minimal examples of statistics required for human genomic analysis. ↩

Odds ratios, SE & CI

Sun, 04 Jul 2021 00:00:00 +0200

Odds ratios, SE & CI

Introduction
Odds ratio
- Perspective 1
- Perspective 2
Standard error
Confidence interval
References

Introduction

Altman & Bland review the use of odds ratio (OR), standard error (SE), and confidence interval (CI) with some examples in (Bland & Altman, 2000); the 42nd paper on the list of statistical notes in their BMJ series, ( lit-altman_bland.md ): 42. Bland JM, Altman DG. (2000) The odds ratio. 320, 1468. ¹

In reproducing their examples I use $X$ and $Y$;

X = Eczema
Y = Hay fever

As an example dataset, they cite the following table; Association between hay fever (Y) and eczema (X) in 11 year old children.

	Y Yes	Y No	Y Total
X Yes	141	420	561
X No	928	13 525	14 453
X Total	1069	13 945	15 522

Odds ratio

Perspective 1

The probability that a child with X will also have Y is estimated by the proportion $\dfrac{141}{561}$ (25.1%) and odds is estimated by $\dfrac{141}{420}$.

Similarly, for children without X the probability of having Y is estimated by $\dfrac{928}{14 453}$ (6.4%) and the odds is $\dfrac{928}{13 525}$.

They compare the groups in several ways:

By the difference between the proportions $\dfrac{141}{561} - \dfrac{928}{14 453} = 0.187$ (or 18.7 percentage points).
The ratio of the proportions (also called the relative risk) $\dfrac{ \left(\dfrac{141}{561}\right) }{ \left(\dfrac{928}{14 453}\right) } = 3.91$.
The OR $\dfrac{ \left(\dfrac{141}{420}\right) }{ \left(\dfrac{928}{13 525}\right) } = 4.89$.

Perspective 2

Looking at the table the other way round, What is the probability that a child with Y will also have X?

The proportion is $\dfrac{141}{1069}$ (13.2%) and the odds is $\dfrac{141}{928}$.

For a child without Y, the proportion with X is $\dfrac{420}{13 945}$ (3.0%) and the odds is $\dfrac{420}{13 525}$.

Comparing the proportions this way,

The difference is $\dfrac{141}{1069} - \dfrac{420}{13 945} = 0.102$ (or 10.2 percentage points);
The ratio (relative risk) $\dfrac{ \left(\dfrac{141}{1069}\right) }{ \left(\dfrac{420}{13 945}\right) } = 4.38$;
The OR $\dfrac{ \left(\dfrac{141}{928}\right) }{ \left(\dfrac{420}{13 525}\right) } = 4.89$.

The OR is the same whichever way round we look at the table, but the difference and ratio of proportions are not. This is because the two OR are

$\dfrac{ 141 / \textbf{420} }{ \textbf{928} / 13 525 }$ and $\dfrac{ 141 / \textbf{928} }{ \textbf{420} / 13 525 }$ which can both be rearranged to give $\dfrac{ 141 \times 13 525 }{ 928 \times 420 }$.

Swapping orders for rows and columns produces the same OR.

Swapping orders for either only rows or only columns produces the the reciprocal of the OR, $1/4.89 = 0.204$.

Thus, OR can indicate the strength of the relationship. OR cannot be negative but is not limited in the positive direction, producing a skew distribution. Reversing the order of categories for one variables simply results in a reversed sign of log OR:

$log(4.89) = 1.59$,

$log(0.204) = - 1.59$.

Standard error

The standard error (SE) can be calculated for the log OR and hence a confidence interval (CI).

The SE of log OR is simply estimated by the square root of the sum of the reciprocals of the four frequencies. For the example,

\[\text{SE(}log \text{OR)} = \sqrt{ \frac{1}{141} + \frac{1}{420} + \frac{1}{928} + \frac{1}{13 525}} = 0.103\]

Confidence interval

A 95% confidence interval (CI) for the log OR is obtained as 1.96 standard errors on either side of the estimate.

For the example, the log OR is $log_{e} (4.89) = 1.588$ and the confidence interval is $1.588 \pm 1.96 \times 0.103$, which gives $1.386$ to $1.790$.

The antilog of these limits to give a 95% CI for the OR itself, as $exp(1.386) = 4.00$ to $exp(1.790) = 5.99$.

The observed OR, 4.89, is not in the centre of the confidence interval because of the asymmetrical nature of the OR scale. For this reason, in graphs ORs are often plotted using a logarithmic scale. The OR is 1 when there is no relationship. We can test the null hypothesis that the OR is 1 by the usual ${\chi}^2$ test for a two by two table.

Despite their usefulness, ORs can cause difficulties in interpretation. Altman & Bland review this debate and also discuss ORs in logistic regression and case-control studies in future Statistics Notes.

References

Bland, J. M., & Altman, D. G. (2000). The odds ratio. Bmj, 320(7247), 1468.

Receiver operating characteristic plots

Fri, 16 Jul 2021 00:00:00 +0000

Receiver operating characteristic plots

Receiver operating characteristic plots
Example ROC cuvre
References

Receiver operating characteristic plots

This article covers the fifth paper in the series of statistics notes (Altman & Bland, 1994) ( lit-altman_bland.md ): 5. Altman DG, Bland JM. (1994) Diagnostic tests 3: receiver operating characteristic plots. 309, 188, and concerns quantitative diagnostic tests. ¹ Diagnosis based on yes or no answers are covered in another note by Bland and Altman. The same statistical methods for quantifying yes or no answers can be applied here when there is a cut off threshold for defining normal and abnormal test results. For simplicity, I will call someone who is diagnosed by a clinical test a “case” and someone who is not diagnosed by a test/healthy/normal, a “control”. These terms are incorrect but much simpler to repeatedly read than “people who are diagnosed by a test”.

The receiver operating characteristic (ROC) plot can be used measure how test results compare between cases and controls. Altman and Bland mention that this method was developed in the 1950s for evaluating radar signal detection. An aside for history buffs, from wikipedia:

The ROC curve was first used during World War II for the analysis of radar signals before it was employed in signal detection theory.[56] Following the attack on Pearl Harbor in 1941, the United States army began new research to increase the prediction of correctly detected Japanese aircraft from their radar signals. For these purposes they measured the ability of a radar receiver operator to make these important distinctions, which was called the Receiver Operating Characteristic.[57]

The example shown in Figure 1 uses graft versus host disease, with an index measurement whose definition is not important. The Yes indicate cases and No indicate controls in our terminology, respectively. The usefulness of the test for predicting graft versus host disease will clearly relate to the degree of non-overlap between the two distributions. A ROC plot is obtained by calculating the

sensitivity and
specificity of every observed data value and plotting, as in Figure 2,
Y axis = sensitivity,
X axis = 1 - specificity.

A test that perfectly defines cases and cotrols would have a curve that aligns withe Y axis and top. A test that does not work would produce a straight line matching the centerline. In practice, overlaps always occur such that the curve usually lies somewhere between, as shown in Figure 2.

The performance of the test (diagnostic accuracy) is reported as the area under the ROC curve. The area is equal to the probability that a random case has a higher measurement than that of a control. This probability is .5 for a test that does not work (e.g. coin-toss; straight line curve). This discriminatory power assessment is important for a clinical test if it is to be sufficient to discriminate cases and controls.

At this stage we have the global assessment of discriminatory power showing that a test can divide cases and control. A cut off for clinical use also requires a local assessment. As per Altman and Bland; the simple approach of minimising “errors” (equivalent to maximising the sum of the sensitivity and specificity) is not necessarily best. We must consider any type of costs of

false negatives
false positives
and prevalence of disease in the test cohort.

In their example:

cancer in general population
- most cases should be detected (high sensitivity)
- many false positives (low specificity), who could then be eliminated by a further test.

For comparing two or more measures, the ROC plot is useful. The curve wholly above another is clearly the better test. Altman and Bland cite a review for methods for comparing the areas under two curves for both paired and unpaired data.

In my (reccomended) pocket-sized copy of Oxford handbook of medical statistics (Peacock & Peacock, 2011), a clinical example uses a chosen cut-off of sensitivity $>81\%$ and specificity $28\%$. The area under ROC curve was .65, thus a moderately high predictive power. The accuracy (proportion of all correctly identified cases) was $\frac{ 30 + 42 }{ 185 } = 39\%$

\[\frac{\text{No. cases above cutoff} + \text{No. controls below cutoff }}{ \text{cohort total} }\]

Example ROC cuvre

To implement this method, I include here an example in R code.

# Modified example from https://stackoverflow.com/questions/31138751/roc-curve-from-training-data-in-caret

library(caret)
library(mlbench)

# Dataset
data(Sonar)
# An example dataset for classification of sonar signals using a neural network. The task is to train a network to discriminate between sonar signals bounced off a metal cylinder and those bounced off a roughly cylindrical rock. Each pattern is a set of 60 numbers in the range 0.0 to 1.0. Each number represents the energy within a particular frequency band, integrated over a certain period of time. Labels: "R" if the object is a rock and "M" if it is a mine (metal cylinder). 

ctrl <- trainControl(method="cv", 
                     summaryFunction=twoClassSummary, 
                     classProbs=T,
                     savePredictions = T)

rfFit <- train(Class ~ ., data=Sonar, 
               method="rf", preProc=c("center", "scale"), 
               trControl=ctrl)

library(pROC)
# Select a parameter setting
selectedIndices <- rfFit$pred$mtry == 2

# Plot:
plot.roc(rfFit$pred$obs[selectedIndices],
         rfFit$pred$M[selectedIndices], 
         print.auc=TRUE,  
         print.thres=TRUE, #thresh .557 shown
         legacy.axes=TRUE)

# With ggplot
library(ggplot2)
library(plotROC)
ggplot(rfFit$pred[selectedIndices, ], 
       aes(d=( ifelse(rfFit$pred$obs[selectedIndices]=="M",1,0) ),
           m = M 
           )) + 
  geom_roc(hjust = -0.4, vjust = 1.5) + 
  coord_equal() +
  style_roc() + 
  annotate("text", x=0.75, y=0.25, label=paste("AUC =", round((calc_auc(g))$AUC, 4)))

References

Altman, D. G., & Bland, J. M. (1994). Diagnostic tests 3: receiver operating characteristic plots. BMJ: British Medical Journal, 309(6948), 188.
Peacock, J., & Peacock, P. (2011). Oxford handbook of medical statistics. Oxford university press.

Sensitivity and specificity

Mon, 19 Jul 2021 00:00:00 +0200

Sensitivity and specificity

Sensitivity and specificity
Confidence intervals
Quantifying the diagnostic ability
References

Sensitivity and specificity

The third paper on the list of BMJ statistics notes by Altman and Bland, ( lit-altman_bland.md ), (Altman & Bland, 1994) 3. Altman DG, Bland JM. (1994) Diagnostic tests 1: sensitivity and specificity. 308, 1552. ¹

The simple diagnostic test such as an x-ray is used to classify patients into two groups:

Presence of a symptom or sign
- Yes
- No

Altman and Bland use the following cited example; The results of a scan (test) compared to the correct diagnosis (true positive) based on either necropsy, biopsy, or surgical inspection. i.e. How good is the scan for correct diagnosis?

Table 1. Relation between results of liver scan and correct diagnosis.

Liver scan	: Pathology (diagnosis) :
	Abnormal (+)	Normal (-)	Total
Abnormal(+)	231	32	263
Normal(-)	27	54	81
Total	258	86	344

Patients who are correctly labelled are:

Disease signs and abnormal liver
- 258 true positives
No signs and healthy liver
- 86 true negatives

The proportions of these two groups that were also correctly diagnosed by the scan were $231/258=0.90$ and $54/86=0.63$, respectively.

Sensitivity
- Proportion of true positives that are correctly identified by the test.
Specificity
- Proportion of true negatives that are correctly identified by the test.

Based on Altman and Bland’s example sample, we expect 90% true positives (patients with abnormal pathology to have abnormal (positive) liver scans), and 63% true negatives (those with normal pathology would have normal (negative) liver scans).

Confidence intervals

The sensitivity and specificity are proportions, so confidence intervals can be calculated. This uses standard methods for proportions (Gardner & Altman, 1989).

Quantifying the diagnostic ability

Sensitivity and specificity are one approach to quantifying the diagnostic ability of the test. In this case, we already have the final results of tests and diagnosis from the sample set. For an individual patient we only have the test result. We want to quantify how well the test can predict true positives.

This is answered in the next statistical note; predictive values. It defines positive and negative predictive values and requires the use of sensitivity, specificity, and prevalence.

References

Altman, D. G., & Bland, J. M. (1994). Diagnostic tests. 1: Sensitivity and specificity. BMJ: British Medical Journal, 308(6943), 1552.
Gardner, M. J., & Altman, D. G. (1989). Calculating confidence intervals for proportions and their differences. Statistics with Confidence. London: BMJ Publishing Group, 28–33.

Bayesian statistics introduction

Sat, 22 Apr 2023 00:00:00 +0000

Bayesian statistics introduction

Bayes’ Theorem Formula
Bayesian and Frequentist Methods: Two Statistical Philosophies
- Bayesian Approach
- Frequentist Approach
Bayesian Methods: An Overview and Examples
- How Bayesian Methods Work
  - Example 1: Estimating Prevalence
  - Example 2: Using Bayes’ Theorem to Update Estimates
- Bayesian Terminology
Prior Distributions in Bayesian Methods
Likelihood and Posterior Distributions in Bayesian Methods
Summarizing and Presenting Bayesian Results
Using Bayesian Analyses in Medicine
- Example: GREAT – A Bayesian Re-analysis
Using Bayesian Analyses
- Paroxetine and Suicide Attempts: A Bayesian Analysis
- Posterior Distributions
Bayesian Checklist
- ROBUST (Reporting Of Bayes Used in Clinical STudies)
Comparison of Bayesian and Frequentist Methods
References

This material is reproduced based on the examples from: Oxford handbook of medical statistics (Janet Peacock Philip J Peacock) [1].

Bayes’ Theorem Formula

Describes the Bayesian approach to statistical analysis in contrast to the frequentist approach
Covers prior and posterior distributions, their roles, and choices
Discusses examples, pros and cons of Bayesian methods in medicine, and how to interpret them in medical literature
Developed by Reverend Thomas Bayes in 1763
Statement about conditional probabilities, used in many statistical areas
Events A and B:
$Pr(A \mid B)$: ‘the probability of A happening given that B has already happened’
$Pr(A \mid B)$ = $\frac{Pr(B \mid A) \times Pr(A)}{Pr(B)}$

Bayesian and Frequentist Methods: Two Statistical Philosophies

Bayes’ theorem forms the basis for competing statistical philosophies: Bayesian and frequentist methods
They differ in their definitions of probability

Bayesian Approach

Probability interpreted as degree of belief that an event will occur
Degree of belief comes from past data or experience
Unknown quantities (e.g., means, proportions) follow probability distributions expressing certainty about true value
Degree of belief can be updated with further information

Frequentist Approach

Probability is the long-run frequency of events (r) that occur in n trials
Probabilities estimated directly from samples
Unknown quantities (e.g., means, proportions) considered fixed, estimated from data with confidence intervals

Bayesian Methods: An Overview and Examples

How Bayesian Methods Work

Combines anticipated values or distribution of values (prior) with new data
Updates the distribution of the true values

Example 1: Estimating Prevalence

Specify anticipated value or distribution for prevalence (prior) using national data
Collect regional data and calculate prevalence
Combine observed area prevalence with prior distribution to update the distribution of true prevalence in the region

\[\text{Prior} + \text{New data} = \text{Updated estimate}\]

Example 2: Using Bayes’ Theorem to Update Estimates

Study investigated new D-dimer test for diagnosing venous thromboembolism (VTE) in patients with suspicious symptoms [3]
Calculate updated probability that a patient has VTE given a positive D-dimer test result: $\Pr(VTE+ \mid D+)$

Using Bayes’ theorem:

$Pr(VTE^{+})$ = anticipated prevalence of VTE = 14% (0.14)
$Pr(D^{+})$ = proportion who test positive on D-dimer = 32% (0.32)
$Pr(D^{+} \mid VTE^{+})$ = probability of positive D-dimer test if the patient truly has VTE = 79% (0.79, sensitivity)

\[Pr(VTE+ \mid D+) = \frac{Pr(D+ \mid VTE+) \times Pr(VTE+)}{Pr(D+)} = \frac{0.79 \times 0.14}{0.32} = 0.346 = 34.6\%\]

Updated probability that a patient testing positive on D-dimer has VTE is approximately 35%
Bayesian approach updates estimates, arguably providing better estimates

Bayesian Terminology

Prior beliefs: Prior distribution
New data: Likelihood
Updated estimate: Posterior distribution

Prior Distributions in Bayesian Methods

Introduction to Prior Distributions

Prior distribution: distribution of unknown quantity combined with new data to provide updated estimate
Three categories of prior distribution [4]:
1. Frequency distribution based on past data
2. Objective representation of reasonable beliefs about a quantity
3. Subjective measure of an individual’s beliefs

Common ‘Default’ Prior Distributions

Non-informative/reference priors: e.g., uniform distribution; used when range of values can be pre-specified but no clear opinion about which value is most likely
Informative-sceptical prior: used to express scepticism about the estimated quantity, reduces chances of spuriously large effects
Informative-enthusiastic prior: counterbalance of sceptical prior; used when positive effect is expected
Informative prior based on prior beliefs, which are formally elicited: shape of distribution varies by context, Normal distribution sometimes used

Sensitivity Analyses

Choice of prior distribution can affect final estimate
Good practice to test sensitivity of assumptions by using different forms
If choice doesn’t affect updated estimate, results are more robust
If choice matters, present range of results to demonstrate sensitivity to the prior

Key Points on Prior Distributions [5]

Choice is based on judgement, subjectivity is unavoidable
Use a range of options to test sensitivity of choice
Clearly justify choice(s) of prior to make results credible to external consumers

Likelihood and Posterior Distributions in Bayesian Methods

Likelihood

Summary of evidence from new data
Combined with prior distribution to create updated posterior distribution

Posterior Distribution

Updated probability distribution for unknown quantity
Reflects range of possible values and degree of belief associated with each value
Less uncertainty than prior distribution; tends to be narrower than prior distribution and likelihood

Example: Combining Prior Distribution and Data

Prior distribution: evidence available before study
Likelihood: evidence from study itself
Posterior distribution: pools two sources of evidence by multiplying curves together [6]
Prior distribution pulls likelihood towards null value (0), making final result less extreme

Figure 1. Illustration of how Bayesian analysis combines a prior distribution (top graph) with the data (‘likelihood’, middle graph) to give the posterior distribution (bottom graph). Reproduced from BMJ, Spiegelhalter et al, 319, 508 1999.

Conjugate Distributions

Common for prior and posterior distributions to be related (same distribution or family of distributions)
Makes calculations more feasible

Summarizing and Presenting Bayesian Results

Estimates

Summary measure (mean, median) often presented
Other estimates (standard deviation, interquartile range) depend on distribution shape and context

Posterior Probabilities

Posterior distribution is a probability distribution
Can calculate probabilities for a specific range of values for estimated quantity
Example: Probability that relative risk in a trial is greater than 1 (showing efficacy)

Credible Intervals (Posterior Interval)

Commonly present 95% credible intervals for posterior estimates
Represents range within which true value lies with 95% probability
Different from 95% confidence intervals (based on sampling distribution, not probability distribution)
Easy to calculate for unimodal and symmetrical posterior probability distributions; more complex otherwise

95% Credible Interval vs. 95% Confidence Interval

95% probability that true value lies within 95% credible interval
95% probability that a 95% confidence interval contains true value
Differences between the two are subtle and may not be practically significant for data interpretation

Significance Tests

No formal place in Bayesian framework
Emphasis on distribution of estimates, not testing against a single value
Posterior probability distribution can be used to calculate probability that true value takes specific values
Bayesian approach provides information needed, rather than yes/no approach of significance testing
Both Bayesian and frequentist statisticians agree that single value or test against single value is of limited usefulness

Using Bayesian Analyses in Medicine

Bayesian methods are used in various medical research areas:

Observational studies
Design, monitoring, and analysis of trials
Meta-analyses
Missing data imputation
Decision making
Health economics

Applicable in situations similar to frequentist methods (single quantity estimation, regression analysis, multifactorial regression, multilevel models)

Example: GREAT – A Bayesian Re-analysis

Original GREAT trial examined effect of thrombolytic therapy on patients with suspected myocardial infarction
Frequentist analysis reported significant beneficial effect on mortality
Results challenged due to unexpected size, small trial, and more modest effect in unpublished bigger European trial
Pocock and Spiegelhalter conducted Bayesian re-analysis, constructing prior distribution to express plausible reduction in mortality
Bayesian analysis showed a reduction in mortality of 25% compared to 49% in frequentist analysis, suggesting over-optimistic trial results
Demonstrates how Bayesian analysis can be used to combine prior evidence and new data for more reasonable estimates in cases of unexpected results

Using Bayesian Analyses

Paroxetine and Suicide Attempts: A Bayesian Analysis

Meta-analysis examined the potential link between antidepressant drugs and increased suicides in adults [10]
Included unpublished data, corrected for duration of medication and placebo treatment
7 suicide attempts in patients taking the drug, 1 in patient taking placebo
Prior distribution assumed to be gamma
Three different prior distributions tested: pessimistic, slightly pessimistic, and slightly optimistic (Fig. 2)

Figure 2. Three different prior distributions used in a meta-analysis of antidepressant drugs and suicide in adults. From Aursnes et al. [10]

Posterior Distributions

Posterior distributions correspond to the three prior distributions
Majority of each distribution is greater than 0, indicating evidence in favor of an adverse effect of paroxetine on suicide risk
Authors reported relative risks of 2.46 (pessimistic prior), 2.20 (slightly pessimistic prior), and 2.34 (optimistic prior)
Bayesian approach supports recent meta-analyses results, suggesting increased risk of suicidal activity in adults taking certain antidepressant drugs

Figure 3. Three posterior distributions corresponding to the three priors used in a meta-analysis of antidepressant drugs and suicide in adults. From Aursnes et al. [10]

Bayesian Checklist

Sung et. al have generated a checklist of seven items (ROBUST) that should be included when a Bayesian analysis is reported[11]. These are helpful in interpreting a Bayesian analysis.

ROBUST (Reporting Of Bayes Used in Clinical STudies)

The box below lists the items included in ROBUST. The checklist can be scored to provide a measure of the quality of reporting, but here it is given as a guide to what points to check when reading an article where Bayesian methods have been used.

Prior distribution: specified
Prior distribution: justified
Prior distribution: sensitivity analysis
Analysis: statistical model
Analysis: analytical technique
Results: central tendency
Results: standard deviation or credible interval

Comparison of Bayesian and Frequentist Methods

This table is adapted from Spiegelhalter et al. and gives a helpful summary of the two approaches.

Issue	Frequentist methods	Bayesian methods
Prior information other than that in the study being analysed	Interpretation of the parameter of interest: Informally used when choosing a model/form of analysis, A fixed unknown value	Used formally by specifying a prior probability distribution, An unknown quantity which can have a probability distribution
Basic statistical question	How likely are the data, given a particular value of the parameter?	How likely is the particular value of the parameter given the data?
Presentation of results	P values, estimates, confidence intervals	Plots of posterior distribution of the parameter, calculation of specific posterior probabilities of interest, and use of the posterior distribution in formal decision analysis. Expected value and credible intervals
Dealing with subsets in trials	Adjusted P values (e.g. Bonferroni)	Subset effects shrunk towards zero by a ‘sceptical’ prior

Adapted from Spiegelhalter et al. 1999.

References

Oxford handbook of medical statistics (Janet Peacock Philip J Peacock).
Bayes T. An essay towards solving a problem in the doctrine of chances. Philos Trans Roy Soc 1763; 53:370–418.
Kovacs MJ, Mackinnon KM, Anderson D, O’Rourke K, Keeney M, Kearon C et al. A comparison of three rapid D-dimer methods for the diagnosis of venous thromboembolism. Br J Haematol 2001; 115(1):140–4.
Ashby D. Bayesian statistics in medicine: a 25 year review. Stat Med 2006; 25(21):3589–631.
Spiegelhalter DJ, Abrams KR, Myles JP. Bayesian approaches to clinical trials and health-care evaluation. Chichester, West Sussex: Wiley, 2004.
Spiegelhalter DJ, Myles JP, Jones DR, Abrams KR. Methods in health service research: An introduction to Bayesian methods in health technology assessment. BMJ 1999; 319(7208):508–12.
Gelman A, Carlin John B, Stern Hal S, Rubin DB. Bayesian data analysis. 2nd ed. Boca Raton, FL: Chapman & Hall/CRC, 2004.
GREAT group. Feasibility, safety, and efficacy of domiciliary thrombolysis by general practition- ers: Grampian Region Early Anistreplase Trial. GREAT Group. BMJ 1992; 305(6853):548–53.
Pocock SJ, Spiegelhalter DJ. Domiciliary thrombolysis by general practitioners. BMJ 1992; 305(6860):1015.
Aursnes I, Tvete IF, Gaasemyr J, Natvig B. Suicide attempts in clinical trials with paroxetine randomised against placebo. BMC Med 2005; 3:14.
Sung L, Hayden J, Greenberg ML, Koren G, Feldman BM, Tomlinson GA. Seven items were identified for inclusion when reporting a Bayesian analysis of a clinical study. J Clin Epidemiol 2005; 58(3):261–8.

Burden testing with variant collapse

Mon, 26 Apr 2021 00:00:00 +0000

Burden testing with variant collapse

Introduction
To do
Main papers in order
- Major Classes of Tests
- Burden Tests So Far
  - Tests
  - Power of burden tests depends on
Li and Leal 2008
Madsen and Browning 2009
Price 2010
- Methods
- Results and Discussion
Neale 2011
- C-alpha tests
SKAT
Lee 2012
- Introduction
- Methods
Protein pathway analysis
- Zhang 2021 ASHG
- Problems with this paper
  - Cluster method
  - Carrier counts instead of allele dosage
Itan & Casanova contributions
- Historical review
Burden test power calculation
References

Introduction

In this series I will go over methods and theory for gene burden testing. I am publishing this live as I work since there are several heavy papers to get through.

To do

Running code on test data: Remake the gVCF as BP_RESOLUTION to list a site for every position. Filter: If variants are MAF >1.1 in gnomAD then run a filter to remove variants that are not sequenced in controls. Justification: If we ran that filter for every singleton it would remove the very rare variants that would have been a true positive if it had been sequenced in the controls.

Methods for variant collapse: https://www.nature.com/articles/s41576-019-0177-4/ (Povysil et al., 2019) One of the most important starting positions in our analysis is determined on variant collapse. Each project and biological system has unique features that dertermine how variants should be grouped for joint analysis. Some more context is available in this blogpost https://lawlessgenomics.com/2021/05/28/pathway_analysis.html.

Main papers in order

Methods for detecting associations with rare variants for common diseases: application to analysis of sequence data (Li & Leal, 2008).
A groupwise association test for rare mutations using a weighted sum statistic. (missing reference).
An evaluation of statistical approaches to rare variant analysis in genetic association studies. (Morris & Zeggini, 2010).
Pooled association tests for rare variants in exon-resequencing studies. (Price et al., 2010).
Testing for an unusual distribution of rare variants. (Neale et al., 2011).
Rare-Variant Association Testing for Sequencing Data with the Sequence Kernel Association Test. (missing reference).
Optimal tests for rare variant effects in sequencing association studies. (Lee et al., 2012).
Optimal Unified Approach for Rare-Variant Association Testing with Application to Small-Sample Case-Control Whole-Exome Sequencing Studies. (Lee et al., 2012).
Sequence Kernel Association Tests for the Combined Effect of Rare and Common Variants. (Ionita-Laza et al., 2013)

Major Classes of Tests

Burden/Collapsing tests
Supervised/Adaptive Burden/Collapsing tests
Variance component (similarity) based tests
Omnibus tests: hedge against difference scenarios

Burden Tests So Far

Tests

Binary Collapsing: CAST
CMC
Count Collapsing: MZ (GRANVIL)
Weighted Sum Test

Power of burden tests depends on

Number of associated variants
Number of non-associated variants
Direction of the effects.
Powerful if most variants are causal and have effects in the same direction.

Li and Leal 2008

Methods for detecting associations with rare variants for common diseases: application to analysis of sequence data (Li & Leal, 2008)
The collapsing method, which involves collapsing genotypes across variants and applying a univariate test, is powerful for analyzing rare variants.
The multivariate analysis is robust against inclusion of non-causal variants.
Both methods are superior to analyzing each variant individually with univariate tests.
Combined Multivariate and Collapsing (CMC) method unifies the advantages of both collapsing and multiple-marker tests.

Background

Genotype phenotype association may be tested by:

A single market test (i.e. GWAS) may use standard univariate statistical tests (e.g., chi-squared test, Fisher’s exact test, or Cochran Armitage test for trend) and with the family-wise error rate (FWER) controlled by a multiple-comparison correction (e.g., Bonferroni, permutation).
A multiple-marker test,
- which tests multiple variant sites simultaneously with the use of multivariate methods,
- such as the Fisher product method,
- Hotelling’s T test or logistic regression.
Collapsing variants can increase power.
Classification of variants can be a major problem.
If classified correct power of causal detection is higher than single- and multiple-marker testing.
If miss-classified, non-functional variants could introduce error.

Genetic model

In a locus M variants can independently cause disease.
A = allele
1-M = allele label
p = allele frequency
Gk = 0/1/2 genotype
Fki = penertance of geno for ith variant.
Relative risk of each variant i in models: additive, multiplicative, dominant, and recessive.
Prevalence of disease caused by each individual variant is calculated.
Prevalence of disease caused by entire locus can therefore be calculated.
Total prevalence can be approximated by the sum of the individual prev.
Same pheno can be due to different causal variants.
The proportion of cases as a result of variant i can be ascertained.
These cases are members of group i.
There is a total of M groups.
For group i the frequency of variant i can also be produced for 0,1,2.
Then, the expected freq of genotype G at variant i across all M groups of cases can be found.
Controls expected genotype freq at i variant can be found.
The expected freqs are used to calculate the power to detect assoc in this study.
Their focus is the omnibus test, which provides an association test of the entire locus and is not focused on any specific variant within the locus.

Single-marker test

One approach of association studies is to test each variant site individually with the use of a univariate test and assess the significance of the omnibus test (chi-squared test or F test) after correction for multiple comparisons.
For univariate tests, a contingency table can be constructed to compare genotype frequencies at each variant site in cases and controls.
Because an observation of individuals that are homozygous for the high-risk rare allele is extremely rare, AA genotypes are collapsed with Aa genotypes, and a table is constructed.
For an equal number of cases and controls, the classical Pearson chi-squared statistic for testing equal genotype frequencies in cases and controls is used.

Multiple-marker test

Another option is to test all variants simultaneously by multivariate test

e.g. Fisher product method, Hotelling’s T-squared test, or multiple logistic regression.
(This is not the same as one regression run for every single variant with multiple covariates, as done in GWAS sometimes).
The authors give an example using Xiong et. al., (2002). Generalized T2 test for genome association studies. Am. J. Hum. Genet.
Note: Hotellings T-squared tests the differences between the (multivariate) means of different populations, where tests for univariate problems would make use of a t-test. The distribution is named for Harold Hotelling, who developed it as a generalization of Student’s t-distribution.
Rejecting the null hypothesis would indicate that at least one of the variants is assocciated with disease.

Basic collapsing method

Since variants are expected to be very rare, genotypes across all variants are collapsed. An individual is coded once if they have any one variant for the locus.

1 = rare variants present.
0 = no rare variants. The classic Pearson chi-squared statistic can be used to test the summed proportion of variants in cases vs controls. The power can be calculated as shown by the authors.

CMC method

Unified method that combines collapsing and multivariate testing.

Markers are divided into subgroups (e.g. allele frequency).
Within group, markers are collapsed.
A multivariate test is then used (Hotelling’s T-squared).

e.g. M markers in a locus are split into K groups. There are n markers in group g, and so on for each group. The multivariate test is done where for each group, the individuals are coded with either 1 (a carrier of one or more variants) or 0 (wild-type).

Power of CMC method

Results:

Freq below 0.01 collapsed.
Freq above 0.01 not collapsed.
With misclassification, power is much better with CMC method compared to collapsing method
- particularly when with a high frequency of the non-causal variant.
Slight (very slight I think) loss of power when causal variants are high frequency compared to collapsing.
- This scenario is unlikely otherwise single-marker test may be possible to detect.

Misclassification

Two types of miss-classifications are considered:

inclusion of non-functional variants and
exclusion of functional variants.
There is a large section on results of Excluding Functional Variants and Inclusion of Nonfunctional Variants.

This section can be read for detail but is probably familiar enough for us to skip.

Effects of Linkage Disequilibrium (LD)

A simulation is done in this section:

The locus has six variants, with a total allele frequency of 0.05.
Four of the variants have an allele frequency of 0.01 and are on different haplotypes.
Each of the remaining two variants, with allele frequencies of 0.005, is on one of the haplotypes where a variant with allele frequency of 0.01 resides;
- there is complete LD between these variants (r 2 z0:5).
For comparison purposes, a second simulation was carried out, in which all variants were on separate haplotypes.
One thousand replicates were generated, and the power was evaluated for an a level of 0.001.

Results:

In the presence of LD, the power for the single-marker test, Hotelling’s T2 test, and the collapsing method is
- 0.075, 0.63, and 0.85, respectively.
Data generated with each variant on a separate haplotype, the corresponding powers are
- 0.011, 0.451, and 0.737, respectively.

Evaluation of Type I Error Rate

Simulation was used to generate data under the null hypothesis of no association between variants and disease status.
This process was repeated for 5000 replicates.
It was then evaluated whether or not each replicate had a p value % 0.05.
The type I error rate was estimated by the proportion of replicates with a p value below 0.05.
A type I error rate above 0.05 signifies a higher false-positive rate, and conversely, a type I error rate below 0.05 indicates a conservative test.

Result:

Hotelling’s T2 test and the collapsing method:
- well controlled and slightly conservative.
Logistic regression for multiple-marker test and the likelihood-ratio test performed on the basis of an asymptotic chi-squared distribution:
- not well controlled.
- The inflation increases with decreasing allele frequencies.
CMC method, with multivariate Hotelling’s T2 test or logistic regression:
- well controlled.

Problems with this paper

The main problem with collapsing method is when single individuals have more than one causal variant being only counted as “1”. This more likely to occur for our protein-pathway collapse than a single gene/locus collapse.
Reason 1: dosage is not counted. Someone may have homozygous and only counts for 1 rather than 2 (or some other weight).
Reason 2: someone with more than one variant is only counted as 1 (if it is in the same allele frequency group for collapse). This may be rare in a single gene but is more likely in a protein network collapse. Usually it would be OK particularly if the second variant is in LD with the first and therefore should only be counted once.
CMC code - maf threshold: In the R code, those variants with minor allele frequency below the specified maf threshold are collapsed into a single super variant. Can we group data into several frequency groups or is one above/below threshold only possible?

Madsen and Browning 2009

A groupwise association test for rare mutations using a weighted sum statistic (Madsen & Browning, 2009)

Background

CAST: Cohort allelic sums test - an existing grouping method in which the number of individuals with one or more mutations in a group (e.g. gene) is compared between affected and unaffected individuals.
Cites three papers:
- [Cohen JC, Kiss RS, Pertsemlidis A, Marcel YL, McPherson R, et al. (2004) Multiple rare alleles contribute to low plasma levels of HDL cholesterol. Science 305: 869–872.]
- [Li and Leal 2008]
- [Morgenthaler S, Thilly WG (2007) A strategy to discover genes that carry multi- allelic or mono-allelic risk for common diseases: A cohort allelic sums test (CAST). Mutation Research/Fundamental and Molecular Mechanisms of Mutagenesis 615: 28–56.]
CMC: Combined Multivariate and Collapsing - the alternative method for grouping variants. All rare variants are collapsed, as in the CAST method, and the collapsed variants are treated as a single common variant which is analysed together with the other common variants using multivariate analysis. In the CMC version used in [Li and Leal], rare variants are defined as those having a minor allele frequency (MAF) of at most 1%.
In this study, they presume a group of candidate functional variants.
Propose a weighted-sum method in which mutations are
- grouped according to function (e.g. gene),
- and each individual is scored by a weighted sum of the mutation counts.
To test for an excess of mutations in affected individuals, they use permutation of disease status among affected and unaffected individuals.
- Using permutation adjusts for the weighting of the mutations and the requirement that a mutation must be observed to be included in the study.

Note that permutation of disease status results in correct type I error even in the presence of LD [Cheverud 2001, Churchill 1994], although relatively low LD is expected between rare variants [Li and Leal 2008, Pritchard 2001,2001].

Main theory

The weighted-sum method deviates from the CAST method [5,27]: weighting the variants differently when determining the genetic load of an individual. By weighting the signals from each mutation, the weighted sum method accentuates mutations that are rare in the unaffected individuals, so that the test is not completely dominated by common mutations.
In the CAST method,
- common variants have a high impact on the group signal,
- and if many common mutations are present in a group, almost all individuals will have one or more mutations.
- To avoid this effect a threshold on the mutation-frequencies may be used, as suggested in the CMC method [Li and Leal 2008].
- May be difficulty to select biological meaningful threshold, affecting test outcome.
In the weighted-sum method,
- include mutations of all frequencies,
- but mutations are weighted according to their frequency in the unaffected individuals.

Methods

Weighted-Sum Method

Compares the number of mutations in a group of variants between case/control. Designed to identify an excess of mutations in case, compared to controls. Each variant belongs to a group (gene, pathway, etc.) and, for a group with L variants, the method is comprised of the following steps:

(A)
- For each variant, choose ALT/minor allele
- define number in cases/controls.
- calculate the weight for a variant.
- It is used to down-weight mutation counts in constructing the weighted-sum score; see (B) and (C) below
Eqn 1: Weight
- qi = ( ((no. mut allele in control) + 1)/(2(no. controls) +2) )
- Wi = root (total samples x qi(1-qi))
- For 10 case (7 mut), 10 control (5 mut), weight for variant is 0.447
Mutation freq of controls is used so that true signal in cases is not deflated by using to total number.
A drawback when variance of scaled mut-freq, hence loss of power when mutation freq is high.
The one and two to the numerator and denominator, respectively, avoids zero estimates - used on next steps.
(B)
- Genetic score of each individual calculated.
- j - Genetic score of individual.
- Iij - number of mutations for variant i in individual j.
- Generic model uses 0/1/2.
- Recessive model only homozygous are assigned 1.
- Dominant model both het and homo are assigned 1.
(C)
- All sample combined are ranked according to genetic scores.
- The sum of ranks for cases is calculated.
- Under null-hypothesis, normal distribution of random variables using a ranking procedure equivalent to that in Wilcoxon test.
(D)
- Case/control status is permuted
- steps (A)-(C) repeated k times to sampled X1,..Xk.
(E)
- The average and sample standard deviation of X1,…,Xk are calculated and standardized score-sum is found (z).
- Under the null hypothesis, this has an approximately standard normal distribution.
- Thus, a p-value for association test can be obtained by comparing z in the quantiles of standard normal.
An alternative p-value calculation is also provided using a standard permutation with a stopping rule (slighly slower for their power similations). That also gives uniform (0,1) distributed p-value, which might be useful for further analysis.
Permutation of case/control labels maintains the LD structure.

Power simulations

Comparison of:

Weighted-sum method,
CAST,
CMC,
and variant-by-variant methods,

For each set of parameters,

100 datasets are simulated,
the four methods are applied,
and the proportions of significant outcomes used as the power estimates.

To mimic a genome wide study of about 20,000 fairly independent human genes, we calculate a p-value for each gene, and use a significance threshold of 0.05/20000 = 2.561026 in all power simulations.

Genetic Models:

Recessive, Additive and Dominant models:
- the disease-related variants act independently,
Recessive-Set model:
- outcome of a mutation at one variant depends on the presence of a mutation at another variant.

Frequency spectra:

Wright’s formula used to sample the unaffected population frequency spectrum of the mutations at each variant for the Recessive, Additive and Dominant models. e.g. The mutation probability is calculated such that the probability (pM) that a haplotype contains at least one disease-risk mutation is fixed in unaffected individuals.

A method for sampling is provided and ignored in this summary.

A method is provided to compare models with damaging variants and benign variants.

NB. Tested Variants.

The mutation probabilities (p) can be very low for some of the sampled variants. This means that some variants contain no mutations in any of the sampled individuals, and these variants are hence omitted in the tests.
ENCODE data

The method was tested on rare variants with the frequency-spectrum of a naturally occurring population using the ENCODE resequencing data.

Results

Proportion of Variants Containing Mutations

The mutation frequencies are sampled according to Wright’s formula, and hence mutations are very rare for some variants. Using 1000 affected and 1000 unaffected individuals, mutations are on average observed at only 49.4% of the variants (sd: 4.9%). This level is in concordance with the level from human resequencing studies.

Skipping the section: Power under Varying Model Parameters.

Discussion

Analysis of pathways can be done in two different ways. One way is to use the pathway as a group, and run the test on the entire pathway. On the other hand, for large pathways, it may be beneficial to use a method that allows a gene with a strong signal to have a high impact on the combined pathway test-statistic (T). If a pathway contains G non-overlapping genes, a method to do this is to use the weighted-sum method on each gene, and combine the resulting p-values (p1,…,pG) with the Fisher product test statistic.

Summary

Weight based on frequency of variat in the controls.
Sum of ranks in cases.
Case/control labels permuted n times, mean, SD, standardised score-sum found (z).
p-value for association test obtained by comparing z in the quantiles of standard normal.
Alternative p-value using a standard permutation with a stopping rule gives uniform (0,1) distributed p-value, which might be useful for further analysis.

Price 2010

Pooled association tests for rare variants in exon-resequencing studies. (Price et al., 2010)
The approach is based on the regression of
- phenotypic values
- on individuals’ genotype scores
- subject to a variable allele-frequency threshold,
- incorporating computational predictions of the functional effects of missense variants.
Statistical significance is assessed by permutation testing with variable thresholds.

Methods

Simulation Framework

Simplified summary:

Model of European ancestry to recapitulate the site-frequency spectrum of non-synonymous human SNPs.
9kb for 10K individuals.
Missense only.
Phenotypes
- quantitative normal distribution
- carrier and non-carrier same variance
- carriers shifted by $\delta$ SD

Weighted Approaches Correspond to Implicit Assumptions about Log Odds Ratios

(missing reference) - Based on the frequency in controls. - Relationship between log odds ratio and allele frequency the same as Madsen 2009.

Fixed-Threshold Approach

(missing reference)

i indexes SNPs,
Ci is the reference allele count of SNP i in cases,
and xi ($\xi_i$) is equal to 1 if the frequency of SNP i is below a specified threshold (1% or 5%)
and is equal to 0 otherwise.
They generalise this for quantitative phenotypes.
- add the term: pj ($\pi_j$) - the phenotype of sample j. - I need to compare these two papers here in more detail (genetic score section). (missing reference)
- Generic model uses 0/1/2.
- Recessive model only homozygous are assigned 1.
- Dominant model both het and homo are assigned 1.

Weighted Approach

(missing reference)

Also generalise by adding $\pi_j$ for a quantitative phenotype value.

Variable-Threshold Approach

There exists some (unknown) threshold T for which variants with a minor allele frequency (MAF) below T are substantially more likely to be functional than are variants with an MAF above T.
z-score of a regression across samples of phenotypes versus counts of mutations meeting the allele-frequency threshold T.

Cheating Approach to Incorporating $\varphi(p)$ (“phi(p)”)

Weight variants according to the probability $\varphi(p)$
- that an allele of frequency p is functional,
- as inferred by using the same simulated data used to evaluate power.

Incorporation of Computational Predictions of Functional Effects

Tested if incorporation of PolyPhen-2 scores improves statistical test.
To asses, simulated PolyPhen-2 predictions of damaging and neutral mutations.

Application to Empirical Data Sets

Applied to some real data.

Results and Discussion

Skipping this.
The methods are an improvement. (missing reference)
and additionaly adds a method for the quantitative variable.

Neale 2011

Testing for an unusual distribution of rare variants. (Neale et al., 2011)

C-alpha tests

They propose here the C-alpha test statistic as a novel approach for testing for
- the presence of this mixture of effects across a set of rare variants.
Unlike existing burden tests, C-alpha, by testing the variance rather than the mean, maintains consistent power when the target set contains both risk and protective variants.

SKAT

Rare-Variant Association Testing for Sequencing Data with the Sequence Kernel Association Test (missing reference)

Introduction

They propose the sequence kernel association test (SKAT),
- regression method to test for
- association between genetic variants (common and rare)
- in a region
- and a continuous or dichotomous trait
- while easily adjusting for covariates.
As a score-based variance-component test, SKAT can
- quickly calculate p values analytically by
- fitting the null model containing only the covariates,
- and so can easily be applied to genome-wide data.
A limitation for previous burden tests is that they implicitly assume that all rare variants influence the phenotype in the same direction and with the same magnitude of effect (after incorporating known weights).
However, one would expect most variants (common or rare) within a sequenced region to have little or no effect on phenotype,
whereas some variants are protective and others deleterious,
and the magnitude of each variant’s effect is likely to vary (e.g., rarer variants might have larger effects).
Collapsing across all variants is likely to introduce substantial noise.
Burden tests require either specification of thresholds for collapsing
or the use of permutation to estimate the threshold.

Methods

Sequencing Kernel Association Test

*SKAT is a supervised test - for the joint effects of multiple variants - in a region - on a phenotype. -

For each region, SKAT analytically calculates a p value for association while adjusting for covariates.
Adjustments for multiple comparisons are necessary for analyzing multiple regions,
for example with the Bonferroni correction or FDR control.

Notation:

n subjects
p variants
covariates
$y_i$ phenotype for sample i
$X_i$ covariats
$G_i$ genotypes (0,1,2 general)

SKAT Model and Test for Linear SNP Effects

Eqn 1 and 2:

linear regression and
logistic regression

Eqn 3:

variance-component score statistic
The weight is adjusted based on the MAF
- if rarer variants are expected to be more likely to have larger effects.
- still putting decent nonzero weights for variants with MAF 1%–5%.
A special case of SKAT arises when the outcome is dichotomous, no covariates are included, and all $W_j$ = 1.
Under these conditions, we show in Appendix A that the SKAT test statistic Q is equivalent to the C-alpha test statistic T.

Relationship between Linear SKAT and Individual Variant Test Statistics

One needs to fit the null model only a single time to be able to compute the Sj for all individual variants j as well as all regions to be tested.
Similarly, if multiple regions are under consideration, then the same mb0 can be used to compute the SKAT Q statistics for each region.

Accommodating Epistatic Effects and Prior Information under the SKAT

Ability to model the epistatic effects of sequence variants on the phenotype within the flexible kernel machine regression framework.

They replace Gi’b by a more flexible function f(Gi) in the linear and logistic models (1) and (2) where f(Gi) allows for rare variant by rare variant and common variant by rare-variant interactions

SKAT Is a Generalization of the C-Alpha Test

The recently proposed the C-alpha test has advantages over burden tests in that
- it explicitly models the possibility that minor alleles can be deleterious or protective.
However, it does not currently allow for the analysis of quantitative outcomes or the inclusion of covariates and p value calculation requires permutation.
They demonstrate that for a dichotomous trait in the absence of covariates,
- the C-alpha test statistic is equivalent to the SKAT statistic with unweighted linear kernel,
- which is the same as the kernel machine test in Wu et al.

Lee 2012

Introduction

SKAT-O

Optimal tests for rare variant effects in sequencing association studies. (Lee et al., 2012)
A class of tests that include burden tests and SKAT as special cases, and derive an optimal test within this class that maximizes power.

Methods

They show the SKAT method
They show the buden testing methods
The show a method for:
- $Q_p$ = (1 - $p$)$Q_{SKAT}$ + $pQ_{burden}$
- The resulting optimal test corresponds to a best linear combination of SKAT and burden tests that maximizes power.

Protein pathway analysis

Zhang 2021 ASHG

Zhang et al. provide and excellent example of the same problem that we are assessing. (Zhang et al., 2021) A computational approach for detecting physiological homogeneity in the midst of genetic heterogeneity

I have never had a paper so similar to a project that we are currently working on. Basically the only difference is:

the cluster size method (theirs might work better, using total edge weights instead of protein count),
cluster separation by NHC instead of MCL,
using KEGG+REACTOME to additionally define the description of the cluster (but this can be done at the end).
The method in main analysis is carrier count instead of SKAT style (we have compared both).

Same cohort size. Really nice to see it in ASHG. Will have to do a comparison to per-gene SKAT-O to show the improvement, as they do.

Problems with this paper

Cluster method

I surprised to see them apply their own novel clustering method (NHC) without some formal methods paper first - so many already exist, including 10 that can be used with their String data. In theory it sound good but there is no real evidence.

Carrier counts instead of allele dosage

Their main analysis is:

(a) with PC-adj = glm(Pheno ~ Carrier-status + PCs, family=binomial).
- The information for carrier status is missing, presumably 0/1 resulting in a sort of burden test.
(b) without PC-adj = fisher exact test.
- When doing this we got great results with for our earlier tests but thought it might be too simplistic.
Using pathway SKAT-O (accounts for frequency/dosage) they got poorer results.

Main problems here:

One count per gene (as data$CARRIER in analysis) presumes that any variant (homozygous or heterozygous, compound het, etc) is deleterious in the individual. In restricted to LoF then OK, but missense variants are included in the study (and will usually be the majority of candidates). Filters are applied to get a high quality list, but there will still be a majority of missense VUS.

In the section “Application to an HSE cohort (V): Comparison with a burden test on the HSE cohort”, SKAT-O is not a burden test; while their method is actually more similar to a burden test since they are labelling an individual if they carry a qualifying variant in a gene. This is just a terminology error but it indicates that maybe the key differences in pathway testing have not been considered, as written up here in our review.

One could argue that only counting once per gene could help prevent problems due to LD. Although, the WES/WGS data would have the phasing information included to prune out LD if that was the argument. Basically, I think this is just a simplification rather than an advantage. If SKAT had happened to work better they would have used those results.

e.g. In Fig 3, TLR3 variants = 7 cases, 1 Hom, 9 Het. This seems to amount to a carrier status of 7 in their dataset for the glm and fisher exact.

Their main analysis:

# call R command to run pc-adj enrichment
ro.r("data <- read.table('temp_pc.txt', header=T, sep='\t')")
ro.r("fit <- glm(data$PHENOTYPE ~ data$CARRIER+data$PC1+data$PC2+data$PC3, family='binomial')")
ro.r("adjusted.pval <- anova(fit, test='LRT')[2, 5]")
r_pvalue = ro.r("adjusted.pval")
pvalue = r_pvalue[0]
pvalue = float('%.3E' % Decimal(pvalue))
file_out.write(str(pvalue) + '\t')

# pathway enrichment
this_case_pathway_hit = dict()
for each_pathway in pathway_gene_set_dict.keys():
	pathway_gene_set = pathway_gene_set_dict[each_pathway]
	case_pathway_overlap = this_case_gene_set & pathway_gene_set
	case_in = len(case_pathway_overlap)
	case_out = len(this_case_gene_set) - case_in
	pathway_in = len(pathway_gene_set)
	pathway_out = len(pathway_genes) - pathway_in
	if case_in != 0:
		odd, pvalue = stats.fisher_exact([[case_in, case_out], 
		[pathway_in, pathway_out]], alternative='two-sided')
		adj_pvalue = pvalue * pathway_size
		if adj_pvalue < 0.00001:
			adj_pvalue = float('%.3E' % Decimal(adj_pvalue))
			this_case_pathway_hit[each_pathway] = adj_pvalue

# They do not show any code with the SKAT parameters, presumably defaults using R.

Itan & Casanova contributions

Historical review

We are tackling this problem of protein pathway analysis from the viewpoint of rare immune disease and infection. Historically, several topics in bioinformatic and functional analysis have been required before we could achieve our current position of statistically-robust genetic discovery for rare disease:

Candidate variant select for individual genomes
Compiling reliable cohorts of patients with shared phenotypes
Protein-protein interactions
Variant collapse
Protein pathway analysis
Functional validation

Therefore, a historical review of the timeline is beneficial to illustrate the technical successes that allow us to reliably produce candidate variants by genome sequencing and to validate statistically-driven results by “traditional” functional validation. These steps [1, 2 and 6 in our list] are exemplified by the following historical review. The complete list of steps 1-6 are touched on, but full validation of each step is the culmination of what we are currently working on and will be explicitly reviewed when we have completed our study.

As one of the leaders in this field, Casanova lab has provided a lot of insider history to the story in a great twitter thread https://twitter.com/casanova_lab/status/1397539593608695808. The literature for discussion is first listed here to facilitate downloading but sources are referenced as usual throughout.

2013 PNAS. The human gene connectome as a map of short cuts for morbid allele discovery. (Itan et al., 2013) https://pubmed.ncbi.nlm.nih.gov/23509278/

2014 BMC Gen. HGCS: an online tool for prioritizing disease-causing gene variants by biological distance. (Itan et al., 2014) https://pubmed.ncbi.nlm.nih.gov/24694260/

2015 Front. Novel primary immunodeficiency candidate genes predicted by the human gene connectome. (Itan & Casanova, 2015) https://pubmed.ncbi.nlm.nih.gov/25883595/,

2015 PNAS. The human gene damage index as a gene-level approach to prioritizing exome variants. (Itan et al., 2015) https://pubmed.ncbi.nlm.nih.gov/26483451/

2016 NatMet. The mutation significance cutoff: gene-level thresholds for variant predictions. (Itan et al., 2016) https://pubmed.ncbi.nlm.nih.gov/26820543/

2015 PNAS. Can the impact of human genetic variations be predicted? (Itan & Casanova, 2015) https://pubmed.ncbi.nlm.nih.gov/26351682/

2018 Bioinf. PopViz: a webserver for visualizing minor allele frequencies and damage prediction scores of human genetic variations. (Zhang et al., 2018) https://pubmed.ncbi.nlm.nih.gov/30535305/

2019 PNAS. Blacklisting variants common in private cohorts but not in public databases optimizes human exome analysis. (Maffucci et al., 2019) https://pubmed.ncbi.nlm.nih.gov/30591557/

2019 NAR. SeqTailor: a user-friendly webserver for the extraction of DNA or protein sequences from next-generation sequencing data. (Zhang et al., 2019) https://pubmed.ncbi.nlm.nih.gov/31045209/.

2020 Hum Gen. The human genetic determinism of life-threatening infectious diseases: genetic heterogeneity and physiological homogeneity? (Casanova & Abel, 2020) https://pubmed.ncbi.nlm.nih.gov/32462426/

2021 JCI. Herpes simplex encephalitis in a patient with a distinctive form of inherited IFNAR1 deficiency. (Bastard et al., 2021) https://pubmed.ncbi.nlm.nih.gov/32960813/

2021 JCI. TLR3 controls constitutive IFN-β antiviral immunity in human fibroblasts and cortical neurons. (Gao et al., 2021) https://pubmed.ncbi.nlm.nih.gov/33393505/

2021 AJHG. A computational approach for detecting physiological homogeneity in the midst of genetic heterogeneity (Zhang et al., 2021) https://pubmed.ncbi.nlm.nih.gov/34015270/

To date, the main paper that implements protein pathway analysis for rare immune disease is that by Peng Zhang and Yuval Itan (Zhang et al., 2021).

The history begins with determining methods for candidate variant selection - the main challenge in human genomics, especially for individual patients who can benefit from precision medicine. Over the last decade, we have reached a point where we can now reasonably discern individual candidate-causal variants from the background noise of genomic variability.

From the authors’ perspective, the project began around 2011 and its first step was concluded in 2013 with Yuval Itan’s first “Human Gene Connectome” paper while he was a post-doc with Casanova lab. (Itan et al., 2013). This software connected genes like streets in a map, based on their physiological relatedness. It was soon followed by methodological development (Itan et al., 2014) and application to inborn errors of immunity (Itan & Casanova, 2015) or both (Itan et al., 2015), and a couple of necessary detours (Itan et al., 2016) and (Maffucci et al., 2019). A review was also written by two of the main authors during the same period (Itan & Casanova, 2015).

When Peng Zhang joined the Casanova lab as post-doc, Yuval Itan had started his own lab. However, the pair worked together to continue producing the papers on variant interpretation and data processing (Zhang et al., 2018) and (Zhang et al., 2019).

After completing this period of work, they renamed “Human Gene Connectome II” the “Network-based Heterogeneity Clustering”. At this point, their aims were defined as being generally indistinguishable from ours. That is, “the detection of physiological homogeneity in a cohort of patients sharing a clinical phenotype but with high genetic heterogeneity - a hallmark of severe infectious diseases” (Casanova via twitter), as presented in their next paper on this topic (Casanova & Abel, 2020).

Shen-Ying Zhang came on board as senior author on the next two papers. With an excellent database of immune disorders and infections, the team could gradually build their software. Exomes from patients with HSV-1 encephalitis were used for testing successive versions in (Bastard et al., 2021) and (Gao et al., 2021).

Quoting Casanova “When they were capable of detecting the known TLR3-IFN needles in the HSE stack, they installed camp 1, rested a bit and reported to me on the radio, while I was watching them from the basecamp with binoculars. I encouraged them to push for the final ascent and they did.”

With the same goal as our own - producing unbiased methods for detection of biologically-connected causal genetic variation - they found new gene variants that interact via the TLR3-IFN protein pathway, in individual patients. Shen-Ying Zhang found them to be biochemically deleterious, an important factor for validation of genetic-first aproaches. In this case, Zhang et al get as close to the “gold-standard” as anyone to date.

The functional validation of candidate variants in disease then provided a proof-of-principle indication that they could detect physiological homogeneity in the midst of genetic heterogeneity (Zhang et al., 2021).

Quoting Casanova “A computational approach for detecting physiological homogeneity in the midst of genetic heterogeity. That was terrific!”.

An aside on what I call the “gold-standard” for our field should be:

Unbiased statistical detection of a genetic phenomenon.

Validation by functional models under systematic control.

Part [1] Depends on patient cohorts that are large enough to detect the effect based on the phenotype strength - difficult for rare disease.
Part [2] Depends on independently testing biological mechanisms.

This second step generally consists of two hurdles:

If the same researchers perform (1) genetic stats and (2) functional work, there is a bias that is difficult to avoid when trying to functionally validate statistically positive results.

If the statistical genetic associations happen to contain a false positive for something like severe rare immune disease, the sensitive functional models may detect a damaging response. One might find a truly damaging biological mechanism, but if the statistical genetic association is not correct then this biological mechanism should not be deemed causal; back-tracking at this stage would be very difficult due to self-imposed bias.

Ideally, in the future we hope to see a separation of the two steps (stat genetics and wet-lab) such that each are performed independently. The wet-lab would also ideally focus their routines on a particular protein pathway/system with SOPs that improve accuracy and precision (e.g. clinical diagnostics labs, clinical trials) rather than setting up models for each new study.

Returning to our historical review, we have been producing our protocols similarly in parallel. With patient cohorts of comparable sizes and phenotypes we will soon have a complementary validation of protocols. However, great care is also being taken to test and select the most reliable statistical methods for association testing - an improvement to the fine work by (Zhang et al., 2021).

Best practices in candidate variant selection protocols are basically standardised as of 2021, so the main remaining task is standardisation of the protein-pathway annotation and association testing methods - steps which we will soon be ready to publish after peer-review.

Burden test power calculation

This code seems to work
# This script tests then the minimum number of heterozygous mutations that can exist in a test set to reach significance.

# Required inputs, can be modified to read data variable
# SAMPLES (equal case and control)
# SNPs (the number of SNPs in the test pathway, can modify to loop every pathway)
# CMC settings may need adjustment to match your model

library(AssotesteR)

# Pathway size of 5 SNPs
# 100 case and 100 controls

# for (SNPs in 1:3) {
 for (SNPs in 10) {

  SAMPLES = 100
  TOTAL = SAMPLES*SNPs

for(i in 1:TOTAL) {
  # cases
    val_1 <- replicate(i,0)
    x <- (TOTAL-i)
    val_2 <- replicate(x,0)
    
    # cases
    val_3 <- replicate(i,0)
    y <- (TOTAL-i)
    val_4 <- replicate(y,1)

    # combine
    mat1.data <-  (c(val_1, val_2, val_3, val_4))
    
    # contols then cases
    genotype <- matrix(mat1.data,nrow=(SAMPLES*2),ncol=SNPs,byrow=TRUE)
    #print(genotype)
    
    stat <- matrix(genotype, dimnames=list(t(outer(colnames(genotype), rownames(genotype), FUN=paste)), NULL))
    print(paste("for", 
                SAMPLES, "cases and ",
                SAMPLES, "controls:"))
    print(paste("with", SNPs, "candidate pathway SNPs"))
    print(paste(sum(stat),
                 "mutations per", TOTAL, "total nucleotides in cases"))
    print(paste(
                 "0 mutation per", TOTAL, "total nucleotides in controls."
                 ))

    # number of cases
    cases = SAMPLES
    
    # number of controls
    controls = SAMPLES
    
    # total (cases + controls)
    total = cases + controls
    
    # phenotype vector
    phenotype = c(rep(0,controls), rep(1,cases))
    #print(phenotype)
    
    #mycmc = CMC(phenotype, genotype, maf=0.0000001, perm=100)
    #print(mycmc)
    myskat = SKAT(phenotype, genotype, kernel = "linear")
    print(myskat["asym.pval"])
}
}

However, I am filling up SNPs columns so that all cases get the variant mutation. I believe SKAT will give a better score to 5 unique variants rather than 5 shared variants. Therefore, I am currently revising the code.

# SKAT: Sequence Kernel Association Test 
# 
# [1] "for 100 cases and  100 controls:"
# [1] "with 5 candidate pathway SNPs"
# [1] "18 mutations per 500 total nucleotides in cases"
# [1] "0 mutation per 500 total nucleotides in controls."
# $asym.pval
# [1] 0.04799332
# 
# 
# [1] "for 100 cases and  100 controls:"
# [1] "with 10 candidate pathway SNPs"
# [1] "36 mutations per 1000 total nucleotides in cases"
# [1] "0 mutation per 1000 total nucleotides in controls."
# $asym.pval
# [1] 0.04799332

You can view to genotype matrix constructions by uncommenting

 # print(genotype).

Exmple:

[198,]    0    0    0    0    0
[199,]    0    0    0    0    0
[200,]    0    0    0    1    1
[1] "for 100 cases and  100 controls:"
[1] "with 5 candidate pathway SNPs"
[1] "2 mutations per 500 total nucleotides in cases"
[1] "0 mutation per 500 total nucleotides in controls."
$asym.pval
[1] 0.3165047

I will instead try to construct the test set with uniq variants rather than shared. I only realised the problem at the end. An example of what we want instead is - checking the P-value and combinations of variants. Then we will find the real minimum variant count required to pass significance:

# With a small test set
# 2 case, 2 control, 4 positions
# 1 control SNP, 3 case SNP

cases = 2
controls = 2
phenotype = c(rep(0,controls), rep(1,cases))

x <- expand.grid(c(0,1),c(0,1),c(0,1), c(0,1))
geno1 <- x[c(1,2, 3,4),c(1:4)]
geno2 <- x[c(1,2, 9,13),c(1:4)]
geno3 <- x[c(1,2, 1,14),c(1:4)]
geno4 <- x[c(1,2, 1,15),c(1:4)]

geno1
(SKAT(phenotype, geno1, kernel = "linear")["asym.pval"])
geno2
(SKAT(phenotype, geno2, kernel = "linear")["asym.pval"])
geno3
(SKAT(phenotype, geno3, kernel = "linear")["asym.pval"])
geno4
(SKAT(phenotype, geno3, kernel = "linear")["asym.pval"])


# > geno1
#   Var1 Var2 Var3 Var4
# 1    0    0    0    0
# 2    1    0    0    0
# 3    0    1    0    0
# 4    1    1    0    0
# $asym.pval [1] 0.0619688
# 
# > geno2
#    Var1 Var2 Var3 Var4
# 1     0    0    0    0
# 2     1    0    0    0
# 9     0    0    0    1
# 13    0    0    1    1
# $asym.pval [1] 0.0311075
# 
# > geno3
#     Var1 Var2 Var3 Var4
# 1      0    0    0    0
# 2      1    0    0    0
# 1.1    0    0    0    0
# 14     1    0    1    1
# $asym.pval [1] 0.0311075
# 
# > geno4
#     Var1 Var2 Var3 Var4
# 1      0    0    0    0
# 2      1    0    0    0
# 1.1    0    0    0    0
# 15     0    1    1    1
# $asym.pval [1] 0.0311075

References

Povysil, G., Petrovski, S., Hostyk, J., Aggarwal, V., Allen, A. S., & Goldstein, D. B. (2019). Rare-variant collapsing analyses for complex traits: guidelines and applications. Nature Reviews Genetics, 20(12), 747–759. https://doi.org/10.1038/s41576-019-0177-4
Li, B., & Leal, S. M. (2008). Methods for detecting associations with rare variants for common diseases: application to analysis of sequence data. The American Journal of Human Genetics, 83(3), 311–321.
Morris, A. P., & Zeggini, E. (2010). An evaluation of statistical approaches to rare variant analysis in genetic association studies. Genetic Epidemiology, 34(2), 188–193.
Price, A. L., Kryukov, G. V., de Bakker, P. I. W., Purcell, S. M., Staples, J., Wei, L.-J., & Sunyaev, S. R. (2010). Pooled association tests for rare variants in exon-resequencing studies. The American Journal of Human Genetics, 86(6), 832–838.
Neale, B. M., Rivas, M. A., Voight, B. F., Altshuler, D., Devlin, B., Orho-Melander, M., Kathiresan, S., Purcell, S. M., Roeder, K., & Daly, M. J. (2011). Testing for an unusual distribution of rare variants. PLoS Genetics, 7(3), e1001322.
Lee, S., Wu, M. C., & Lin, X. (2012). Optimal tests for rare variant effects in sequencing association studies. Biostatistics, 13(4), 762–775. https://doi.org/10.1093/biostatistics/kxs014
Lee, S., Emond, M. J., Bamshad, M. J., Barnes, K. C., Rieder, M. J., Nickerson, D. A., Christiani, D. C., Wurfel, M. M., & Lin, X. (2012). Optimal Unified Approach for Rare-Variant Association Testing with Application to Small-Sample Case-Control Whole-Exome Sequencing Studies. The American Journal of Human Genetics, 91(2), 224–237. https://doi.org/https://doi.org/10.1016/j.ajhg.2012.06.007
Ionita-Laza, I., Lee, S., Makarov, V., Buxbaum, J. D., & Lin, X. (2013). Sequence Kernel Association Tests for the Combined Effect of Rare and Common Variants. The American Journal of Human Genetics, 92(6), 841–853. https://doi.org/https://doi.org/10.1016/j.ajhg.2013.04.015
Madsen, B. E., & Browning, S. R. (2009). A groupwise association test for rare mutations using a weighted sum statistic. PLoS Genet, 5(2), e1000384.
Zhang, P., Cobat, A., Lee, Y.-S., Wu, Y., Bayrak, C. S., Boccon-Gibod, C., Matuozzo, D., Lorenzo, L., Jain, A., Boucherit, S., & others. (2021). A computational approach for detecting physiological homogeneity in the midst of genetic heterogeneity. The American Journal of Human Genetics.
Itan, Y., Zhang, S.-Y., Vogt, G., Abhyankar, A., Herman, M., Nitschke, P., Fried, D., Quintana-Murci, L., Abel, L., & Casanova, J.-L. (2013). The human gene connectome as a map of short cuts for morbid allele discovery. Proceedings of the National Academy of Sciences, 110(14), 5558–5563.
Itan, Y., Mazel, M., Mazel, B., Abhyankar, A., Nitschke, P., Quintana-Murci, L., Boisson-Dupuis, S., Boisson, B., Abel, L., Zhang, S.-Y., & others. (2014). HGCS: an online tool for prioritizing disease-causing gene variants by biological distance. BMC Genomics, 15(1), 1–8.
Itan, Y., & Casanova, J.-L. (2015). Novel primary immunodeficiency candidate genes predicted by the human gene connectome. Frontiers in Immunology, 6, 142.
Itan, Y., Shang, L., Boisson, B., Patin, E., Bolze, A., Moncada-Vélez, M., Scott, E., Ciancanelli, M. J., Lafaille, F. G., Markle, J. G., & others. (2015). The human gene damage index as a gene-level approach to prioritizing exome variants. Proceedings of the National Academy of Sciences, 112(44), 13615–13620.
Itan, Y., Shang, L., Boisson, B., Ciancanelli, M. J., Markle, J. G., Martinez-Barricarte, R., Scott, E., Shah, I., Stenson, P. D., Gleeson, J., & others. (2016). The mutation significance cutoff: gene-level thresholds for variant predictions. Nature Methods, 13(2), 109–110.
Itan, Y., & Casanova, J.-L. (2015). Can the impact of human genetic variations be predicted? Proceedings of the National Academy of Sciences, 112(37), 11426–11427.
Zhang, P., Bigio, B., Rapaport, F., Zhang, S.-Y., Casanova, J.-L., Abel, L., Boisson, B., & Itan, Y. (2018). PopViz: a webserver for visualizing minor allele frequencies and damage prediction scores of human genetic variations. Bioinformatics, 34(24), 4307–4309.
Maffucci, P., Bigio, B., Rapaport, F., Cobat, A., Borghesi, A., Lopez, M., Patin, E., Bolze, A., Shang, L., Bendavid, M., & others. (2019). Blacklisting variants common in private cohorts but not in public databases optimizes human exome analysis. Proceedings of the National Academy of Sciences, 116(3), 950–959.
Zhang, P., Boisson, B., Stenson, P. D., Cooper, D. N., Casanova, J.-L., Abel, L., & Itan, Y. (2019). SeqTailor: a user-friendly webserver for the extraction of DNA or protein sequences from next-generation sequencing data. Nucleic Acids Research, 47(W1), W623–W631.
Casanova, J.-L., & Abel, L. (2020). The human genetic determinism of life-threatening infectious diseases: genetic heterogeneity and physiological homogeneity? Human Genetics, 139, 681–694.
Bastard, P., Manry, J., Chen, J., Rosain, J., Seeleuthner, Y., AbuZaitun, O., Lorenzo, L., Khan, T., Hasek, M., Hernandez, N., & others. (2021). Herpes simplex encephalitis in a patient with a distinctive form of inherited IFNAR1 deficiency. The Journal of Clinical Investigation, 131(1).
Gao, D., Ciancanelli, M. J., Zhang, P., Harschnitz, O., Bondet, V., Hasek, M., Chen, J., Mu, X., Itan, Y., Cobat, A., & others. (2021). TLR3 controls constitutive IFN-βantiviral immunity in human fibroblasts and cortical neurons. The Journal of Clinical Investigation, 131(1).

Fisher's exact test

Fri, 25 Sep 2020 00:00:00 +0200

Fisher’s exact test

https://www.statology.org/fishers-exact-test/ Statology page

Fisher’s Exact Test is used to determine whether or not there is a significant association between two categorical variables. It is typically used as an alternative to the Chi-Square Test of Independence when one or more of the cell counts in a 2x2 table is less than 5.

Fisher’s Exact Test uses the following null and alternative hypotheses:

H0: (null hypothesis) The two variables are independent.
H1: (alternative hypothesis) The two variables are not independent.

Suppose we have the following 2x2 table:

	Group 1	Group 2	Row Total
Category 1	a	b	a+b
Category 2	c	d	c+d
Column Total	a+c	b+d	a+b+c+d = n

The one-tailed p value for Fisher’s Exact Test is calculated as:

\[p = (a+b)!(c+d)!(a+c)!(b+d)! / (a!b!c!d!n!)\]

This produces the same p value as the CDF of the hypergeometric distribution with the following parameters:

population size = n
population “successes” = a+b
sample size = a + c
sample “successes” = a

The two-tailed p value for Fisher’s Exact Test is less straightforward to calculate and can’t be found by simply multiplying the one-tailed p value by two. To find the two-tailed p value, we recommend using the Fisher’s Exact Test Calculator.

Fisher’s Exact Test: Example

Suppose we want to know whether or not gender is associated with political party preference. We take a simple random sample of 25 voters and survey them on their political party preference. The following table shows the results of the survey:

Sex	Democrat	Republican	Total
Male	4	9	13
Female	8	4	12
Total	12	13	25

Step 1: Define the hypotheses.

We will perform Fisher’s Exact Test using the following hypotheses:

H0: Gender and political party preference are independent.
H1: Gender and political party preference are not independent.

Step 2: Calculated the two-tailed p value.

We can use the Fisher’s Exact Test Calculator with the following input:

	Group1	Group2
Category 1	4	9
Category 2	8	4

One tailed p value: 0.081178
Two-tailed p value is 0.115239

Fisher’s Exact test example

The two-tailed p value is 0.115239. Since this value is less than 0.05, we fail to reject the null hypothesis. We do not have sufficient evidence to say that there is any statistically significant association between gender and political party preference.

Additional Resources

The following tutorials explain how to perform a Fisher’s Exact Test using different statistical programs:

How to Perform Fisher’s Exact Test in R https://www.statology.org/fishers-exact-test-in-r/

Fisher’s Exact Test Calculator https://www.statology.org/fishers-exact-test-calculator/

Upervised learning K-Means clustering

Sun, 23 Apr 2023 00:00:00 +0000

Upervised learning K-Means clustering

Major Requirements
K-Means Algorithm
- Major Requirements
Interpreting the Clusters
- Major Requirements
Selecting the Number of Clusters
- Major Requirements
Evaluating the Number of Clusters
- Major Requirements
Hierarchical Clustering
- Major Advantages

This post is based on reading and figures from: Practical Statistics for Data Scientists 50+ Essential Concepts Using R and Python by Peter Bruce, Andrew Bruce, Peter Gedeck.

Clustering is a technique to divide data into different groups, where the records in each group are similar to one another. A goal of clustering is to identify significant and meaningful groups of data. The groups can be used directly, analyzed in more depth, or passed as a feature or an outcome to a predictive regression or classification model.

K-means was the first clustering method to be developed; it is still widely used, owing its popularity to the relative simplicity of the algorithm and its ability to scale to large data sets. K-means divides the data into K clusters by minimizing the sum of the squared distances of each record to the mean of its assigned cluster. This is referred to as the within-cluster sum of squares or within-cluster SS. K-means does not ensure the clusters will have the same size but finds the clusters that are the best separated.

Major Requirements

Divide data into different groups
Identify significant and meaningful groups of data
Minimize the sum of the squared distances of each record to the mean of its assigned cluster
Use within-cluster sum of squares or within-cluster SS
Find the clusters that are the best separated

The sum of squares within a cluster is given by:

\[SS_k = \sum_{i \in Cluster_k} (x_i - x_k)^2 + (y_i - y_k)^2\]

In clustering records with multiple variables (the typical case), the term cluster mean refers not to a single number but to the vector of means of the variables.

A typical use of clustering is to locate natural, separate clusters in the data. Another application is to divide the data into a predetermined number of separate groups, where clustering is used to ensure the groups are as different as possible from one another.

For example, suppose we want to divide daily stock returns into four groups. K-means clustering can be used to separate the data into the best groupings. Note that daily stock returns are reported in a fashion that is, in effect, standardized, so we do not need to normalize the data.

In R, K-means clustering can be performed using the kmeans function. For example, the following finds four clusters based on two variables—the daily stock returns for ExxonMobil (XOM) and Chevron (CVX):

df <- sp500_px[row.names(sp500_px)>='2011-01-01', c('XOM', 'CVX')]
km <- kmeans(df, centers=4)

The cluster assignment for each record is returned as the cluster component (R):

> df$cluster <- factor(km$cluster)
> head(df)
                      XOM        CVX cluster
    2011-01-03 0.73680496  0.2406809       2
    2011-01-04 0.16866845 -0.5845157       1
    2011-01-05 0.02663055  0.4469854       2
    2011-01-06 0.24855834 -0.9197513       1
    2011-01-07 0.33732892  0.1805111       2
    2011-01-10 0.00000000 -0.4641675       1

The first six records are assigned to either cluster 1 or cluster 2. The means of the clusters are also returned (R):

cluster        XOM        CVX
-0.3284864 -0.5669135
0.2410159  0.3342130
-1.1439800 -1.7502975
0.9568628  1.3708892

Clusters 1 and 3 represent “down” markets, while clusters 2 and 4 represent “up markets.”

As the K-means algorithm uses randomized starting points, the results may differ between subsequent runs and different implementations of the method. In general, you should check that the fluctuations aren’t too large.

In this example, with just two variables, it is straightforward to visualize the clusters and their means:

ggplot(data=df, aes(x=XOM, y=CVX, color=cluster, shape=cluster)) +
  geom_point(alpha=.3) +
  geom_point(data=centers,  aes(x=XOM, y=CVX), size=3, stroke=2)

The resulting plot shows the cluster assignments and the cluster means. Note that K-means will assign records to clusters, even if those clusters are not well separated (which can be useful if you need to optimally divide records into groups).

Figure 1. The clusters of K-means applied to daily stock returns for ExxonMobil and Chevron (the cluster centers are highlighted with black symbols)

K-Means Algorithm

K-means is a clustering algorithm that can be applied to a data set with p variables X1, …, Xp. While the exact solution to K-means is computationally very difficult, heuristic algorithms provide an efficient way to compute a locally optimal solution.

Major Requirements

Specify the number of clusters K and an initial set of cluster means
Assign each record to the nearest cluster mean as measured by squared distance
Compute the new cluster means based on the assignment of records
Iterate steps 2 and 3 until the assignment of records to clusters does not change
Specify an initial set of cluster means by randomly assigning each record to one of the K clusters and then finding the means of those clusters
Run the algorithm several times using different random samples to initialize the algorithm
Use the iteration that has the lowest within-cluster sum of squares to get the K-means result

The sum of squares within a cluster is given by:

\[SS_k = \sum_{i \in Cluster_k} (x_i - x_k)^2 + (y_i - y_k)^2\]

The nstart parameter to the R function kmeans allows you to specify the number of random starts to try. For example, the following code runs K-means to find 5 clusters using 10 different starting cluster means:

syms <- c( 'AAPL', 'MSFT', 'CSCO', 'INTC', 'CVX', 'XOM', 'SLB', 'COP',
           'JPM', 'WFC', 'USB', 'AXP', 'WMT', 'TGT', 'HD', 'COST')
df <- sp500_px[row.names(sp500_px) >= '2011-01-01', syms]
km <- kmeans(df, centers=5, nstart=10)

The function automatically returns the best solution out of the 10 different starting points. You can use the argument iter.max to set the maximum number of iterations the algorithm is allowed for each random start.

Interpreting the Clusters

An important part of cluster analysis can involve the interpretation of the clusters. The two most important outputs from kmeans are the sizes of the clusters and the cluster means.

Major Requirements

Check the sizes of the resulting clusters with km$size
Plot the centers of the clusters with ggplot and gather

You can plot the centers of the clusters using the gather function in conjunction with ggplot:

centers <- as.data.frame(t(centers))
names(centers) <- paste("Cluster", 1:5)
centers$Symbol <- row.names(centers)
centers <- gather(centers, 'Cluster', 'Mean', -Symbol)
centers$Color = centers$Mean > 0
ggplot(centers, aes(x=Symbol, y=Mean, fill=Color)) +
  geom_bar(stat='identity', position='identity', width=.75) +
  facet_grid(Cluster ~ ., scales='free_y')

The resulting plot reveals the nature of each cluster. For example, clusters 4 and 5 correspond to days on which the market is down and up, respectively. Clusters 2 and 3 are characterized by up-market days for consumer stocks and down-market days for energy stocks, respectively. Finally, cluster 1 captures the days in which energy stocks were up and consumer stocks were down.

Figure 2. The means of the variables in each cluster (cluster means).

Selecting the Number of Clusters

The K-means algorithm requires that you specify the number of clusters K. Sometimes the number of clusters is driven by the application, while in other cases a statistical approach can be used.

Major Requirements

Use practical or managerial considerations to determine the number of desired clusters
Use a statistical approach such as the elbow method to find the “best” number of clusters

A common approach, called the elbow method, is to identify when the set of clusters explains “most” of the variance in the data. Adding new clusters beyond this set contributes relatively little in the variance explained. The elbow is the point where the cumulative variance explained flattens out after rising steeply, hence the name of the method.

Figure 3 shows the cumulative percent of variance explained for the default data for the number of clusters ranging from 2 to 15. In this example, there is no obvious elbow point since the incremental increase in variance explained drops gradually. This is fairly typical in data that does not have well-defined clusters. This is perhaps a drawback of the elbow method, but it does reveal the nature of the data.

Evaluating the Number of Clusters

In R, the kmeans function doesn’t provide a single command for applying the elbow method, but it can be readily applied from the output of kmeans. You can evaluate how many clusters to retain by considering the replicability of the clusters on new data and whether the clusters are interpretable and related to a general characteristic of the data.

Major Requirements

Apply the elbow method to determine the optimal number of clusters
Use cross-validation to assess the replicability of the clusters on new data
Consider practical considerations in choosing the number of clusters since there is no statistically determined optimal number of clusters

The algorithm iteratively assigns records to the nearest cluster mean until cluster assignments do not change, and the number of desired clusters, K, is chosen by the user.

Hierarchical Clustering

Hierarchical clustering is an alternative to K-means that can yield different clusters and is more sensitive in discovering outlying or aberrant groups or records. It allows the user to visualize the effect of specifying different numbers of clusters and lends itself to an intuitive graphical display for easier interpretation of the clusters.

Major Advantages

Can yield different clusters than K-means
More sensitive in discovering outlying or aberrant groups or records
Allows visualization of the effect of specifying different numbers of clusters
Intuitive graphical display for easier interpretation of the clusters

Hierarchical clustering involves a dendrogram that shows the relationships between clusters. The user can specify the desired number of clusters by cutting the dendrogram at a specific height.