Abstract
It is well recognized that relationships between variables are not always linear or even monotonic. For example, the expressions of cell-cycle, or circadian clock genes, or the abundance of microbes in a dynamic ecology are not expected to be linear. Furthermore, unknown to the researcher, there may be heterogeneous subgroups or clusters in the data. Researchers may be interested in discovering those clusters and derive an overall measure of association between variables of interest accounting for the different clusters as well as deriving associations within each cluster. Although standard concepts of correlations, such as the Pearson or Spearman, are widely used to describe overall associations, they can be misleading in such situations. As researchers continue to generate complex high dimensional data with hidden substructures or clusters, there is an urgent need for a measure that correctly quantifies associations between variables while agnostically accounting for hidden clusters in the data. Using clustering algorithms which are able to detect hidden clusters and association measures which are suitable for quantifying arbitrary relationships within each clusters, we develop a novel association procedure called CLuster based Association Measures (CLAM) to describe association between pairs of univariate as well as multivariate variables. The method is not limited to any specific form of association and is well-suited for heterogeneous data with hidden clusters, which are common in biomedical research. Performance of CLAM is evaluated using a synthetic data as well as real data from diverse applications, such as fission yeast (S. pombe) cell-cycle genes data, intestinal microbiome data from IBD patients, and three well-known imaging data sets, namely DrivFace data, Landsat data, and COIL data.