Method of reducing dimensionality of a set of attributes used to characterize a sparse data set

摘要

A dimensionality reduction method of generating a reduced dimension matrix data set Dnew of dimension mxk from an original matrix data set D of dimension mxk wherein n>k. The method selects a subset of k columns from a set of n columns in the original data set D where the m rows correspond to observations Ri where i=1, . . . , m and the n columns correspond to attributes Aj where j=1, . . . , n and dij is the data value associated with observation Ri and attribute Aj. The data values in the reduced data set Dnew for each of the selected k attributes is identical to the data values of the corresponding attributes in the original data set. The steps of the method include: for each of the attributes Aj in the original data set D, calculating a value of variance of the data values associated with attribute Aj, where the variance value, Var(Aj), of the attribute Aj is calculated as follows: 1 $\mathrm{Var}\left(\mathrm{Aj}\right)=\left[1/m\right]*\sum _{i=1}^{m}itit{\left(\mathrm{dij}-\mathrm{Mean}\left(\mathrm{Aj}\right)\right)}^2,$ where Mean(Aj) is the mean value of the data values corresponding to attribute Aj; selecting the k attributes having the greatest variance values; and generating the reduced data set Dnew by selecting data values in the original data set D corresponding to the selected k attributes.