validwiki - User contributions [en]

Chemometrics

2020-02-01T15:37:36Z

BryanHanson: /* Connection between PCA, ICA and MCR */

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.<ref>Y. B. Monakhova and S. P. Mushtakova "Methodology of Chemometric Modeling of Spectrometric Signals in the Analysis of Complex Signals" ''Journal of Analytical Chemistry'' vol. 72 no. 2 pgs 119-128 (2017).</ref><ref>P. Ebrahimi, N. Viereck, R. Bro and S. B. Engelsen "Chemometric Analysis of NMR Spectra" in ''Modern Magnetic Resonance'' edited by G. A. Webb, Springer International Publishing (DOI 10.1007/978-3-319-28275-6_20-1) (2017).</ref>
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine_Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''. The article by Winning ''et al.'' provides a comparison of PCA and MCR using 1H 400 MHz spectra of a three component mixture as a data set.<ref>H. Winning, F. H. Larsen, R. Bro and S. B. Engelsen. ''Journal of Magnetic Resonance'' Vol. 190 pgs 26-32 (2008).</ref>

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2019-03-12T15:56:50Z

BryanHanson: /* Typical Steps in a Chemometrics Analysis of NMR Data */ Additional reference.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.<ref>Y. B. Monakhova and S. P. Mushtakova "Methodology of Chemometric Modeling of Spectrometric Signals in the Analysis of Complex Signals" ''Journal of Analytical Chemistry'' vol. 72 no. 2 pgs 119-128 (2017).</ref><ref>P. Ebrahimi, N. Viereck, R. Bro and S. B. Engelsen "Chemometric Analysis of NMR Spectra" in ''Modern Magnetic Resonance'' edited by G. A. Webb, Springer International Publishing (DOI 10.1007/978-3-319-28275-6_20-1) (2017).</ref>
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine_Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2019-03-12T15:51:06Z

BryanHanson: /* Typical Steps in a Chemometrics Analysis of NMR Data */ Inserted new reference.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.<ref>Y. B. Monakhova and S. P. Mushtakova "Methodology of Chemometric Modeling of Spectrometric Signals in the Analysis of Complex Signals" ''Journal of Analytical Chemistry'' vol. 72 no. 2 pgs 119-128 (2017).</ref>
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine_Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Nomenclature

2019-03-07T17:01:09Z

BryanHanson: Unbolded stuff

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== Chemical Shift (δ ) ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===Spin-Spin coupling (J Coupling) ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== CPMG: Carr-Purcell-Meiboom-Gill sequence ===

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
===Accuracy===
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

=== Trueness ===
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

===Bias===
Estimate of a systematic measurement error.<ref name=":0" />

===Linear range===
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

===Measurement uncertainty===
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

===Precision===
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

===Repeatability===
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

===Robustness===
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

===Selectivity===
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

=== Limit of detecion (LOD) ===
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

=== Limit of quantification (LOQ) ===
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

=== Dictionary "ISO/IUPAC <-> ICH" ===
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-07T13:49:15Z

BryanHanson: Added stub for CPMG

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== Chemical Shift (δ ) ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===Spin-Spin coupling (J Coupling) ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== CPMG: '''Carr-Purcell-Meiboom-Gill sequence''' ===

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
===Accuracy===
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

=== Trueness ===
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

===Bias===
Estimate of a systematic measurement error.<ref name=":0" />

===Linear range===
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

===Measurement uncertainty===
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

===Precision===
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

===Repeatability===
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

===Robustness===
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

===Selectivity===
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

=== Limit of detecion (LOD) ===
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

=== Limit of quantification (LOQ) ===
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

=== Dictionary "ISO/IUPAC <-> ICH" ===
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-05T20:37:46Z

BryanHanson: /* Validation terms - as they pertain to NMR */ Moved terms up to Sub-Heading 1 (they were 2 and too small compared to the rest of the page).

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== Chemical Shift (δ ) ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===Spin-Spin coupling (J Coupling) ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
===Accuracy===
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

=== Trueness ===
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

===Bias===
Estimate of a systematic measurement error.<ref name=":0" />

===Linear range===
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

===Measurement uncertainty===
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

===Precision===
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

===Repeatability===
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

===Robustness===
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

===Selectivity===
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

=== Limit of detecion (LOD) ===
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

=== Limit of quantification (LOQ) ===
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

=== Dictionary "ISO/IUPAC <-> ICH" ===
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-05T20:34:20Z

BryanHanson: Removed stray link on "Chemical Shift" which led to talk page.

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== Chemical Shift (δ ) ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===Spin-Spin coupling (J Coupling) ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
====Accuracy====
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

==== Trueness ====
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

====Bias====
Estimate of a systematic measurement error.<ref name=":0" />

====Linear range====
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

====Measurement uncertainty====
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

====Precision====
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

====Repeatability====
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

====Robustness====
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

====Selectivity====
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

==== Limit of detecion (LOD) ====
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

==== Limit of quantification (LOQ) ====
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

==== Dictionary "ISO/IUPAC <-> ICH" ====
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-05T20:18:20Z

BryanHanson: Missed a few

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== [[Talk:Nomenclature#Chemical Shift|Chemical Shift]] (δ ) ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===Spin-Spin coupling (J Coupling) ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
====Accuracy====
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

==== Trueness ====
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

====Bias====
Estimate of a systematic measurement error.<ref name=":0" />

====Linear range====
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

====Measurement uncertainty====
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

====Precision====
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

====Repeatability====
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

====Robustness====
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

====Selectivity====
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

==== Limit of detecion (LOD) ====
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

==== Limit of quantification (LOQ) ====
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

==== Dictionary "ISO/IUPAC <-> ICH" ====
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-05T20:17:11Z

BryanHanson: Finished unbolding terms in section 2. The extra formatting was affecting the auto-generated TOC.

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== [[Talk:Nomenclature#Chemical Shift|Chemical Shift]] '''(δ )''' ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== Dipolar Coupling (Magnetic dipole–dipole interaction) ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== Gyromagnetic Ratio ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== NMR Spectrum ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== Multiplicity ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== Solid-state NMR (SSNMR) ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== Spin ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===[[Talk:Nomenclature#Spin-Spin coupling|'''Spin-Spin coupling''']] '''(J Coupling)''' ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== Spin-Lattice Relaxation ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== Spin-Spin Relaxation ===

=== T1 ===

=== T2 ===

=== T2* ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== Time domain NMR spectroscopy ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
====Accuracy====
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

==== Trueness ====
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

====Bias====
Estimate of a systematic measurement error.<ref name=":0" />

====Linear range====
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

====Measurement uncertainty====
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

====Precision====
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

====Repeatability====
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

====Robustness====
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

====Selectivity====
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

==== Limit of detecion (LOD) ====
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

==== Limit of quantification (LOQ) ====
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

==== Dictionary "ISO/IUPAC <-> ICH" ====
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Nomenclature

2019-03-05T20:14:34Z

BryanHanson: Section 2 I'm removing bold formatting on terms, to be consistent with the rest of the document.

==Introduction ==
This page provides definitions for many of the key terms and phrases used in NMR spectroscopy and context for more general terms with regard to their application within the field of NMR

==Basic NMR terms==

=== Bloch Equations ===
The Bloch equations are a set of coupled differential equations which can be used to describe the behaviour of a magnetization vector under any conditions. When properly integrated, the Bloch equations will yield the X', Y', and Z components of magnetization as a function of time.

=== Boltzmann Distribution ===
The distribution of nuclear spins among their possible energy levels at thermal equilibrium. For a positive magnetogyric ratio, produces population excess (polarization) in the direction of Bo.

=== Bo Static magnetic field ===
The magnetic field about whose direction the nuclear magnetic moment precesses. Its magnitude is expressed in Tesla or as the nominal proton precession frequency (e.g., 14T or 600 MHz).

=== [[Talk:Nomenclature#Chemical Shift|'''Chemical Shift''']] '''(δ )''' ===
A measure of the variation of the effective value of the magnetic field Bo experienced by nuclei due to variation in the electromagnetic environments within the molecule. Usually reported in parts per million of applied field or frequency relative to a resonance in a reference compound specific for the nucleus (eg.,tetramethyl silane (TMS) for 1H) and assigned the unit δ

=== '''Dipolar Coupling (Magnetic dipole–dipole interaction)''' ===
The direct though-space interactions of ''nuclear magnetic moment'' vectors proportional to the inverse cube of internuclear distance. In isotropic solution, they average to zero as a result of diffusion but are present in solid state NMR and their effect on nuclear spin relaxation results are measurable nuclear Overhauser effects (NOEs).

=== '''Gyromagnetic Ratio''' ===
Inherent property of a nucleus; can be either positive (e.g. 1H, 13C) or negative (15N, 29Si). For particular values of Bo and the magnetic quantum number, g determines the energy level of the nucleus.

=== '''NMR Spectrum''' ===
A representation of the NMR data with at least one of its time domains transformed to a frequency domain. It typically involves the Fourier transform of a free induction decay.

=== '''Multiplicity''' ===
Each signal in an NMR spectrum may or may not be split into one or more peaks due to interaction with other magnetic vectors, normally adjacent nuclei. This is called signal '''multiplicity''' and in its simplest form results in simple patterns associated with the 2nI+1 rule (singlet, doublet, triplet, quartet, pentet,etc.) although complex multiplets arise through variation in coupling constants and second order multiplets exist where peak overlap occurs.

=== '''Solid-state NMR (SSNMR)''' ===
The application of NMR to solids, rather than the conventional liquid or solution state NMR applications is characterised by the presence of anisotropic (directionally dependent) interactions. ''Chemical shift anisotropy'' (CSA) and the internuclear ''dipolar coupling'' result in severe line broadening of the corresponding NMR signals and specific hardware and methods are required to perform SSNMR to mitigate some of these issues including Magic Angle Spinning (MAS).

=== '''Spin''' ===
A quantum-mechanical property of a nucleus. In NMR, only nuclei with '''''spin''''' I>0 are observable. The spin of a nucleus is dependent on the numbers and alignments of the spins of its individual protons and neutrons. Nuclei with a spin of ½ generate the simplist NMR spectra and are the most commonly studied.

===[[Talk:Nomenclature#Spin-Spin coupling|'''Spin-Spin coupling''']] '''(J Coupling)''' ===
Effect of the spin state of one nucleus on the energy of another nucleus; responsible for peak splitting. Spin coupling is transmitted through intervening bonding electrons, in contrast to the dipolar mechanism.

=== '''Spin-Lattice Relaxation''' ===

The mechanism by which the component of the magnetization vector along the direction of the static magnetic field reaches thermodynamic equilibrium with its surroundings (the "lattice") and characterised by the spin lattice relaxation time T1. It dictates the time required between individual NMR scans to ensure the system has returned to equilibrium

=== '''Spin-Spin Relaxation''' ===

=== '''T1''' ===

=== '''T2''' ===

=== '''T2*''' ===
Time constant for loss of magnetization in the xy plane. Includes losses due to Bo inhomogeneities as well as spin-spin relaxation.

=== '''Time domain NMR spectroscopy''' ===
Time-domain NMR (TD-NMR) is an alternative to classical NMR run at low magnetic field strengths where data analysis is performed directly on the FID, often through the extraction of relaxation time constants. It can provide a relatively cheap and simple technique for QC analysis.

==Experimental terminology==

=== Acquisition time ===
Duration of the digitization of a free induction decay (FID).

=== Adiabatic pulse ===
A pulse that inverts spins by a slow passage of a chirped pulse through resonance resulting in spins that have different resonance frequencies being inverted at different times. This approach provides very high bandwidth and accurate flip angles with high tolerance to spatial variations in RF field intensity

=== Correlation Time ( <math>\tau_c</math>) ===
Parameter related to the mean time during which a molecule maintains its spatial geometry. For an internuclear vector, c is approximately equal to the average time for it to rotate through an angle of one radian.

=== Echo Time ( TE ) ===
The time between the middle of the first RF pulse and the peak of the spin echo

=== Ernst Angle: ===
In a single-pulse experiment, tip angle giving the best signal/noise ratio for a given combination of spin-lattice relaxation time and repetition rate.

=== Free induction decay ( FID ) ===
Decaying, oscillatory time-domain wave-form generated by nuclei that are subjected to an rf pulse to disturb their equilibrium state and are then allowed to precess "freely" in the absence of rf fields to return to equilibrium.

=== Gated Decoupling: ===
Application of spin decoupling only during selected time periods in a pulse NMR experiment. Used to eliminate either 1H-13C spin coupling or the nuclear Overhauser effect from a 1-D 13C spectrum.

=== Hahn Echo ===
See Spin Echo

=== Heteronuclear Decoupling ===
The removal of Spin-Spin coupling between two different nuclei (e.g. 13C and 1H) by appropriate excitation of the decoupled nucleus, normally performed to simplify spectra or to enhance signal to noise by combining individual components of the NMR signal.

=== Homonuclear Decoupling ===
The removal of Spin-Spin coupling between two signals of the same nuclear isotope. This is normally through selective excitation of specific NMR signals within a spectrum or through pure shift experiments such as PSYCHE.

=== Imaginary Spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time

domain signal; usually the imaginary frequency data is not normally displayed and is used for phase correction the real spectrum. In simple 1D spectra equates to the nmr signals being anti-phase

=== Inversion Time (TI) ===
The time between the inversion pulse and the sampling pulse(s) in an inversion recovery sequence.

=== Larmor frequency ===
The frequency of precession of a nucleus about the direction of Bo; equal to Bo/2

=== Longitudinal Magnetization ===
The Z component of magnetization.

=== Magnetogyric ratio ===
See gyromagnetic ratio

=== Magic Angle ===

=== Magic angle spinning ===
A technique in solid-state NMR spectroscopy which employs this principle to remove or reduce the influence of anisotropic interactions by rapid sample rotation about the magic angle (54.7°), which, much like in the case of solutions, effectively averages all orientation-dependent interactions and allow high resolution spectra.

=== Net Magnetization Vector ===
A vector representing the sum of the magnetization from a spin system.

=== Nyquist Frequency: ===
Highest frequency that can be characterized by sampling at a given rate. Equal to one-half of the

=== Non Uniform sampling (NUS) ===
A technique that acquires only a subset of the data points of high dimensionality experiments and uses reconstruction methods to allow the extraction of complete sets of chemical shift information.

=== Observation frequency ===
The frequency corresponding to the centre of the spectrum window under observation and the nominal frequency used to generate the hard, full spectrum pulses.

=== Phase Cycling ===
Repetition of a pulse sequence changing only the phases of the pulse(s) and the phase-sensitive detector reference. The resultant FID's are then added to suppress undesirable signal components

and/or to produce the desired effect of a pulse sequence (e.g., multiple quantum filter)

=== Phase Correction (Phasing) ===
The Linear combination of the real and imaginary parts of a 1-D spectrum to produce peaks with pure absorption mode shape. Normally described by the automatic or manual setting of zero order and 1st order phase constants.

=== Precession ===
Classical description of the behaviour of nuclear magnetic moments, in which the vectors rotate about the Bo axis at their Larmor frequencies.

=== Pulse Sequence ===
A timed series of RF pulses and magnetic field gradients applied to a spin system to produce a signal whose behaviour gives information about some property of the spin system. This constitutes the fundamental description of a specific NMR experiment.

=== Pulsed Field Gradient PFG ===
A pulsed field gradient is a short, timed pulse with spatial-dependent field intensity. Any gradient is identified by four characteristics: axis, strength, shape and duration. These are fundamental in Diffusion experiments, imaging and an alternative to phase cycling for selection of magnetization.

=== Quadrature Detection ===
Quadrature detection involves the collection of time domain NMR data on both the x and y axes of the rotating frame of reference.

=== Radiation Damping ===
Radiation damping is an affect on intense signals in high field NMR. It occurs when the rotating transverse magnetization of the sample is intense enough to induce large enough EMF in the RF coil strong enough that feedbacks to the sample. This will cause a rotaion of magnetisation back to the +Z axis and result in line broaden of the intense peak. It can also produce other effects including changes to amplitude and phase of adjacent peaks.

=== Real spectrum ===
One of two equally sized blocks of frequency data produced by Fourier transformation of the time domain signal; usually only the real frequency data is displayed and in simple 1D spectra equates to the nmr signals being in-phase

=== Relaxation Reagent ===
A paramagnetic species added to the sample to promote more rapid spin-lattice relaxation and a faster pulse repetition rate.

=== Relaxation Delay ===
The time allocated after each acquisition scan for the system to return to equilibrium. In quantitative NMR this is set to ensure a

=== Repetition Time ===
The repetition time (TR) is the length of time between corresponding consecutive points on repeat scans in an NMR experiment and corresponds to the sum of the pulse sequence, acquisition time and relaxation delay.

=== RF Coil ===
An inductor-capacitor resonant circuit used to set up B1 magnetic fields in the sample and to detect the signal from the sample

=== Spectral Width ===
The frequency range, described in either Hz or PPM, that describes the width of a specific dimension in the NMR spectrum.

=== Spectrum Window ===
A description of the exact frequency band observed in a specific dimension in an NMR spectrum. It is normally described by a combination of the centre frequency (observation frequency) and the spectral width.

=== Spin echo ===
A spin echo is the refocusing of spin magnetisation by a 2 consequetive RF pulses. This is typically used to refocus chemical shift by a 90° TE/2 180° TE/2 sequence.

=== Saturation ===
Partial or complete equalization of the populations of the energy levels of a group of nuclei.It occurs when the rate of input of energy from the applied rf field exceeds the rate of energy dissipation by spin-lattice relaxation. It can occur if rf is applied continuously or when pulse repetition rate is too rapid.

=== Selective Decoupling ===
Spin decoupling by irradiation of a single multiplet with low-power rf. Used to identify coupled proton pairs in a 1-D proton spectrum.

=== Selective Irradiation (Selective Excitation) ===
Application of rf to a single frequency. Often used to excite a nucleus to only one of the possible combinations of spin states of the nuclei coupled to it.

=== Selective Pulse ===
Radio frequency (rf) pulse with a narrow frequency bandwidth (long, low-power pulse) to excite nuclei in a limited chemical shift range.

=== Signal to noise ratio: S/N ===

=== Spin-Spin Relaxation Time (T2) ===
Time constant for the loss of magnetization in the xy plane in the absence of Bo inhomogeneities. Includes T1 relaxation plus chemical and spin exchange, etc.

=== Symmetrization ===
Method of removing artifacts from 2-D contour plots that are symmetrical about the diagonal (HH COSY) or the f1 axis (J-spectra). Spots equidistant from midline are compared and replaced either with their average value or lower value of the two.

=== t1 ===
Time domain arising from the regular increments of the delay period in a 2-D experiment. Time during which the FID is acquired in a 2-D experiment.

=== t1 Noise ===
Streaks of spurious signals in a 2-D spectrum parallel to the f1 axis at the f2 of a strong resonance.

=== Three-Dimensional NMR ===
Three-frequency-domain analog of two-dimensional NMR. Requires data collection as a function of three time domains.

=== Time Domain ===
Manifestation of NMR data with signal amplitude expressed as a function of time. e.g., in the display of an FID.

=== Tip Angle ===
Angle through which the magnetization vector is rotated by the B1 field in pulse NMR.

=== Two-Dimensional (2-D) NMR ===
Spectral method in which data are collected in two time domains: acquisition of the FID (t2) and a successively incremented delay (t1). The resulting data matrix is subjected to two successive FT's to produce a spectrum with two frequency axes, usually either chemical shift/chemical shift (correlation spectroscopy) or chemical shift/J-coupling (J-spectroscopy).

=== Zero Filling ===
Addition of zeros to the end of FIDs to adjust the size of the data, typically to a power of two and commonly used to improve the digital resolution of the transformed spectrum.

==Experiments==
===APT: Attached Proton Test===
The ''Attached Proton Test'' is a simple method for 13C NMR that combines signal enhancement by decoupling with the ability to determine the number of protons that are attached to each carbon atom.

=== COLOC: Correlation through Long-Range Coupling ===
A Heteronuclear (typically 13C) detected long range 2D correlation experiment that has mostly been superseded by the HMBC experiments and its variants. Often used to show ''J'' coupling between Hydrogen and Carbon nuclei 2 and 3 bonds apart.

=== COSY: Correlation Spectroscopy ===
A Homonuclear (typically 1H) 2D correlation experiment. The simplest and original 2D experiment but still one of the most common experiments along with its many variations including those with multiple quantum filters, solvent suppression and optimisation for long range couplings. Often used to show the J coupling between Hydrogen nuclei 2 or 3 bonds apart.

=== DEPT: Distortionless Enhancement by Polarisation Transfer ===
A 1D experiment ( typically 13C) used for determining the presence of primary, secondary and tertiary heteronucei. The DEPT experiment for 13C differentiates between CH, CH2 and CH3 groups by variation of the selection angle parameter (the tip angle of the final 1H pulse): 135° angle gives all CH and CH3 in a phase opposite to CH2; 90° angle gives only CH groups, the others being suppressed; 45° angle gives all carbons with attached protons with the same phase.

=== DIPSI: Decoupling or mixing sequence ===
Decoupling In the Presence of Scalar Interactions- A pulse sequence often used for isotropic mixing in TOCSY experiments due to its efficiency over large bandwidths .

=== DOSY: Diffusion Ordered Spectroscopy ===
In it’s basic form, a pseudo 2D experiment that resolves NMR signals of a 1D spectrum in a second dimension through the diffusion times of their corresponding molecules . This is used to differentiate signals from different molecules based around the effective size of the analytes. The technique relies on spacially encoding the sample through the use of pulsed field gradients

=== DPFGSE excitation: Double Pulsed Field Gradient Spin-Echo excitation ===

=== DPFGSE-NOESY:DPFGSE-TOCSY ===

=== DQF-COSY: Double-Quantum Filtered Correlation Spectroscopy ===
A variant of the COSY experiment with a double quantum filter to remove or at least reduce impact of large singlet peaks on dynamic range and artefacts.

=== Excitation sculpting ===

=== EXSY: Exchange Spectroscopy ===
Essentially the same experiment as a NOESY but for the purposes of observing nuclei exchanging chemical environments during the course of the experiment through conformational or chemical exchange.

=== GARP: Broadband Decoupling Sequence ===
A decoupling sequence used to provide decoupling over a large frequency range. Typically used to remove 13C sidebands in 1H NMR spectra

=== HETCOR: Heteronuclear Shift Correlation (or HETERO-COSY) ===
A directly observed (as opposed to 1H observed) one bond heteronuclear correlation experiment. This has largely been replaced by the indirectly observed HSQC and HMBC experiments due to their greater sensitivity. It does however have the advantage of greater resolution in the heteronuclear dimension.

=== HMBC: Heteronuclear Multiple-Bond Correlation ===
HMBC is similar to the HSQC experiments in that it is a 1H detected heteronuclear corellation experiment. This experiment however correlates 1H and heteronuclei connected via several bonds ( typically 2 to 4). The experiment is optimised for a particular value of the coupling constant between the nuclei

=== HMQC: Heteronuclear Multiple-Quantum Correlation ===
This experiment is very similar to the HSQC. It also correlates 1H to a heteronucleus. It is mostly used for 1H,13C correlations

=== HMQC-TOCSY: Heteronuclear Multiple-Quantum Correlation with additional TOCSY transfer ===
A variant of the HMBC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HOESY: Heteronuclear Overhauser Effect Spectroscopy ===
A heteronuclear variant of the TOCSY experiment showing through space interaction between 1h and a heteronucleus, typically 13C through spin lattice relaxation.

=== HOHAHA: Homonuclear Hartmann-Hahn Spectroscopy ===
An alternate name for the TOCSY experiment

=== HSQC: Heteronuclear Single Quantum Correlation ===
A 1H detected 2D heteronuclear experiment that correlates 1H nuclei with their directly coupled nucleus ( through a single bond)

=== HSQC-TOCSY: Heteronuclear Single Quantum Correlation with additional TOCSY transfer ===
A variant of the HSQC experiment that shows correlations from a heteronucleus ( e.g.13C) to not only its directly bound 1H but also to all other 1H nuclei within the spin system

=== HSQMBC: Heteronuclear Single Quantum Multiple-Bond Correlation ===
These experiments provide pure absorption, antiphase lineshapes for precise, direct measurement of (n)J(C,H) coupling constants.

=== H2BC: Heteronuclear Multiple-Bond Correlation over two bonds ===
A variant of the HSQC and HMBC experiment that almost exclusively correlates 1H nuclei with heteronuclei separated by 2 covalent bonds

=== INEPT: Insensitive Nuclei Enhanced by Polarization transfer ===
A pulse sequence that involves the transfer of nuclear spin polarization from spins with large Boltzmann population differences to nuclear spins of interest with low Boltzmann population differences.[1] INEPT uses J-coupling for the polarization transfer in contrast to Nuclear Overhauser Effect (NOE) which arises from dipolar cross-relaxation. It is a building block for many heteronuclear NMR experiments.

=== INADEQUATE: Incredible natural-abundance double-quantum transfer experiment ===
INADEQUATE is a 2D experiment designed to show homonuclear correlations between low natural abundance nuclei such as 13C. Each coupled pair of nuclei gives a pair of peaks on the INADEQUATE spectrum which both have the same vertical coordinate, which is the sum of the chemical shifts of the nuclei; the horizontal coordinate of each peak is the chemical shift for each of the nuclei separately

=== Inversion recovery ===
Pulse sequence used to determine spin-lattice relaxation times.

=== J-MOD: J-modulated spin-echo ===

=== J-RES: J-Resolved Spectroscopy ===

=== NOE: Nuclear Overhauser Effect ===

=== NOE DIFF: NOE Difference Spectroscopy ===

=== NOESY: Nuclear Overhauser Effect Spectroscopy ===
A 2 dimensional NMR experiment based on the NOE difference experiement, it is typically run as a homonuclear 1H experiment, a NOESY spectrum yields through space correlations via spin-lattice relaxation. NOESY also detects chemical and conformational exchange.

=== PSYCHE: Pure Shift Yielded by Chirp Excitation ===
An experiment generating homonuclear decoupled spectra e.g. a 1H spectrum with no 1H, 1H coupling. This is achieved through the concatenation of partial FIDs generated with a series of swept adiabatic pulses.

=== ROESY: Rotating-Frame NOE Spectroscopy ===
Also been known as CAMELPSIN – Cross-relaxation Appropriate for Minimolecules Emulated by Locked SPINs). It is similar to the NOESY experiment and is useful for determining which signals arise from protons that are close to each other in space even if they are not bonded. A ROESY spectrum yields through space correlations via spin-spin relaxation. ROESY also detects chemical and conformational exchange. It has the advantage over th e NOESY experiment that it’s response does not go through a null at specfiific correlation times and is thus preferred for molecules in the region of a few thousand MWt.

=== STD: Saturation Transfer Difference ===

=== TOCSY: Total Correlation Spectroscopy ===
This experiment is typically run as a 2D homonuclear experiment. It creates correlations between all protons within a given spin system, not just between geminal or vicinal protons as in COSY.

=== TOSS:TOtal Suppression of Spinning sidebands ===

=== TROSY ===

==Validation terms - as they pertain to NMR==
====Accuracy====
Closeness of agreement between a measured quantity value and a true quantity value of a measurand.

Accuracy includes the effect of precision and trueness. Accuracy cannot be given a numerical value but meaurement results are said to be 'more accurate' when the measurement errors are reduced.<ref>EURACHEM Guide "Terminology in Analytical Measurement: Introduction to VIM 3" (TAM): https://www.eurachem.org/images/stories/Guides/pdf/TAM_2011_Final_web.pdf</ref>

==== Trueness ====
Closeness of agreement between the average of an infinite number of replicate measured quantity values and a reference quantity value.<ref name=":0">International Vocabulary of Metrology – Basic and General Concepts and Associated Terms (VIM 3rd edition) JCGM 200:2012: https://www.bipm.org/utils/common/documents/jcgm/JCGM_200_2012.pdf</ref>

====Bias====
Estimate of a systematic measurement error.<ref name=":0" />

====Linear range====
Concentration range over which the intensity of the signal obtained is directly proportional to the concentration of the species producing the signal.<ref name=":1">IUPAC ''Compendium of Chemical Terminology (Gold Book):'' https://goldbook.iupac.org/index.html</ref>

====Measurement uncertainty====
Parameter, associated with the result of a measurement, that characterizes the dispersion of the values that could reasonably be attributed to the measurand.<ref name=":1" />

====Precision====
The closeness of agreement between independent test results obtained by applying the experimental procedure under stipulated conditions. The smaller the random part of the experimental errors which affect the results, the more precise the procedure. A measure of precision (or imprecision) is the standard deviation.

Precision is sometimes misused for accuracy. This problem will be avoided if one recognizes that precision relates only to dispersion, not to deviation from the (conventional) true value. Imprecision has been defined as 'the standard error of the reported value.<ref name=":1" />

====Repeatability====
The closeness of agreement between independent results obtained with the same method on identical test material, under the same conditions (same operator, same apparatus, same laboratory and after short intervals of time).

The measure of repeatability is the standard deviation qualified with the term: `repeatability' as repeatability standard deviation. In some contexts repeatability may be defined as the value below which the absolute difference between two single test results obtained under the above conditions, may be expected to lie with a specified probability.<ref name=":1" />

====Robustness====
The robustness of an analytical procedure is “a measure of its capacity to remain unaffected by small, but deliberate variations in method parameters. Robustness provides an indication of the method’s reliability during normal usage.<ref>EURACHEM Guide "The Fitness for Purpose of Analytical Methods": https://www.eurachem.org/images/stories/Guides/pdf/MV_guide_2nd_ed_EN.pdf</ref>

====Selectivity====
The extent to which other substances interfere with the determination of a substance according to a given procedure.<ref name=":1" />

==== Limit of detecion (LOD) ====
Measured quantity value, obtained by a given measurement procedure, for which the probability of falsely claiming the absence of a component in a material is β, given a probability α of falsely claiming its presence.

IUPAC recommends default values for α and β equal to 0.05.<ref name=":0" />

==== Limit of quantification (LOQ) ====
The limit of quantification is the smallest concentration of an analyte that can be determined quantitatively with a specified precision.

==== Dictionary "ISO/IUPAC <-> ICH" ====
Depending on the accreditation environment, it is necessary to follow different guidelines.

In the pharmaceutical industry, for example, the ICH guidelines <ref>ICH Guideline "Validation of Analytical Procedures: Text and Methology":http://www.ich.org/fileadmin/Public_Web_Site/ICH_Products/Guidelines/Quality/Q2_R1/Step4/Q2_R1__Guideline.pdf</ref>must be followed. Some elementary terms of validation are used differently compared to the ISO/IUPAC <ref name=":1" /> recommendations.

A rough association of the terms is shown in the following table.

{| class="wikitable"
!ISO/IUPAC
!ICH
|-
|Truness
|Accuracy
|-
|Accuracy
| --
|-
|Selectivity
|Specificity
|}

==Further Reading==

If you have any questions and/or concerns, please contact [[User:JWarren|John Warren]] [mailto:John.warren@lgcgroup.com Email].
<references />

Chemometrics

2018-08-05T12:53:28Z

BryanHanson: Removed paragraph on TUCKER3 as it probably isn't useful in the 2D NMR context.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine_Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T20:12:51Z

BryanHanson: You have to use the correct file name if you want it to work Bryan!

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine_Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T20:11:36Z

BryanHanson: Still trying to get the image to appear properly.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]]
The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T20:08:20Z

BryanHanson: Edit source to put Fig 3 in manually. I hope.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3)[[File:Alignment.png|thumb|Figure 3. Results of aligning the wine data set.]].<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

File:Wine Alignment.png

2018-08-03T20:03:48Z

BryanHanson: BryanHanson uploaded a new version of File:Wine Alignment.png

Detail of wine NMR data set showing successful alignment.

File:Wine Alignment.png

2018-08-03T19:59:59Z

BryanHanson: BryanHanson uploaded a new version of File:Wine Alignment.png

Detail of wine NMR data set showing successful alignment.

Chemometrics

2018-08-03T14:25:05Z

BryanHanson: Language polishing.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method in this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed as in PCA, providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T14:19:32Z

BryanHanson: Added 3D array figure to 2D section.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
[[File:3Darray.png|thumb|A stack of 2D NMR spectra can be visualized as a data cube or 3D array. Dimensions are labeled from the NMR point of view, and in parentheses, typical indexing annotations.]]
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Stacks of 2D spectra, each from a different sample, can be visualized as a data cube or 3D array (Figure 7).

Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. As in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, the same number of components is computed for each dimension.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

File:3Darray.png

2018-08-03T14:16:30Z

BryanHanson:

3D array showing stack of 2D spectra.

Chemometrics

2018-08-03T14:12:59Z

BryanHanson: Updating 2D section.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called agglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is one such method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. The result of subjecting a stack of 2D NMR spectra to PARAFAC are three matrices representing scores and loadings, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution. Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. In PARAFAC, each dimension returns the same number of components.

Tucker3 is another generalization of PCA for higher order data. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike PARAFAC, each dimension in TUCKER3 can have a different number of components.

TUCKER1, also known as multivariate image analysis, is the 3D method most similar to PCA. In TUCKER1, no components are computed for the F2 and F1 dimensions. Only the sample dimension is reduced.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T14:01:25Z

BryanHanson: Language polishing con't.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (raw, or measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the resulting smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, abstract new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, after interpretation, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several compounds in the sample. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus incorporates concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is (ideally) a spectrum of a pure compound, and thus interpretation is straightforward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extracted spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-08-03T13:53:09Z

BryanHanson: Language polishing.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created created using the [http://www.models.life.ku.dk/datasets Wine NMR] data set which is freely available.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine spectra are shown in red in the plots, while white wine spectra are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding a suitable solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which also apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of so-called bucketing (also known as binning) steps. The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the potential loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

Chemometrics

2018-06-11T19:23:33Z

BryanHanson: Continued to update figures.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
[[File:FlowChart.png|thumb|Figure 1. Flow chart of typical operations in NMR chemometrics.]]
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created based on the free and open source [http://www.models.life.ku.dk/datasets Wine NMR] data set.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine are shown in red in the plots, while white wines are shown in black. Figure 2 shows an example of each type of wine.
=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
*[[File:Scores.png|thumb|Figure 4. Score plot for the wine data set.]]Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers. Figure 4 shows a score plot for the wine data set.
*[[File:Loadings.png|thumb|Figure 5. Loadings plot for PC1 (upper panel) along with reference spectrum from wine data set.]]Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot. Figure 5 shows the loadings for the first principal component of the wine data set, along with a reference spectrum.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
[[File:HCA.png|thumb|Figure 6. Sample HCA plot using the wine data set.]]
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes. Figure 6 shows an example of HCA using the wine data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

File:HCA.png

2018-06-11T19:22:20Z

BryanHanson:

HCA sample plot

File:Loadings.png

2018-06-11T19:20:39Z

BryanHanson:

Loading plot example

File:Scores.png

2018-06-11T19:18:45Z

BryanHanson:

Score plot example

File:FlowChart.png

2018-06-11T19:15:38Z

BryanHanson:

Flow Chart

Chemometrics

2018-06-11T19:04:12Z

BryanHanson: Working on adding figures on wine NMR data set.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below.
[[File:RedWhite.png|thumb|Figure 2. Typical 1H NMR spectra of red and white wines. Red wines are shown in red. The spectra are dominated by peaks from ethanol, and are expanded vertically to show the smaller peaks.]]
Most of the figures here were created based on the free and open source [http://www.models.life.ku.dk/datasets Wine NMR] data set.<ref>F. H. Larsen, F. van den Berg and S. B Engelson. ''J. Chemometrics'' vol. 20 pgs. 198-208 (2006).</ref> This data set is composed 1H NMR of 40 table wines; however in this work we discarded the two rose wines, leaving only red and white wine spectra for analysis. Red wine are shown in red in the plots, while white wines are shown in black. Figure 2 shows an example of each type of wine. [[File:Fig1_YM_FlowChart.png|thumb|
Figure 1. Typical workflow for NMR chemometrics.
]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software. At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.
===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum. An alternative is the clupa method, which is quite robust (Figure 3).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref>

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

File:RedWhite.png

2018-06-11T18:55:25Z

BryanHanson:

Typical 1H NMR spectra of red and white wines (red wines are shown in red). Spectra are naturally dominated by peaks from ethanol, and are expanded vertically to see the smaller peaks.

Chemometrics

2018-04-10T18:59:43Z

BryanHanson: /* Pre-Processing */ Added wine data set alignment example

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.
[[File:Wine Alignment.png|thumb|Detail of wine data set showing alignment outcome.]]
The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). An alternative is the clupa method, which is quite robust (Figure 4 coming soon).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref> [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email].

File:Wine Alignment.png

2018-04-10T18:58:28Z

BryanHanson:

Detail of wine NMR data set showing successful alignment.

Quantitative NMR

2018-03-03T12:30:31Z

BryanHanson: replaced \centerdot which did not look good with \times in equations

==Regulations==

===EU===

*[https://www.edqm.eu/en/european-pharmacopoeia-9th-edition European Pharmacopeia]
*[[German Arzneimittelbuch|Arzneimittelbuch]]

===US===
*[[US Pharmacopoeia|US Pharmacopoeia]]
===Japan===
*[[Japanese Pharmacopoeia| Japanese Pharmacopoeia]]

==Theoretical Basics of qNMR Analysis==

===Quantification===
The fundamental principle of qNMR is the direct proportionality of the signal intensity <math>I_X</math> in the NMR spectrum to the number of nuclei <math>N_X</math> giving rise to the signal:<math display="block">I_X = K_s \times N_X</math>where <math>K_s</math> is a spectrometer constant that is equal for all resonances in an NMR spectrum. The spectrometer constant <math>K_s</math> is affected by the repetition time, pulse excitation and broadband decoupling.

====Relative Quantification====
In the relative quantification method, the molar ratio between compounds <math>X</math> and <math>Y</math> is calculated by the following equation:

<math>
n_X / n_Y

= I_X / I_Y \times N_Y / N_X </math>

The spectrometer constant, <math>K_s</math> , influences all resonances in one spectrum equally, so the <math>K_s</math> term does not appear in the quantitative calculation.

====Absolute Quantification====
To calculate the absolute content of the main product, by-products, or impurities in a given sample, an internal standard of known concentration is used. Here, the contents are directly calculated from the NMR spectrum by the following equation:

<math>P_X = I_X / I_{IS} \times N_{IS} / N_X \times M_X / M_{IS} \times m_{IS} /m_X \times P_{IS}</math>

where <math>I</math> is the integral value, <math>N</math> represents the number of nuclei giving rise to the signal, <math>M</math> is the molar mass, <math>m</math> is the gravimetric weight and <math>P</math> is the purity of the analyte (<math>X</math> ) and the internal standard (<math>IS</math> ), respectively.

===Reference Standards===
===Parameter and Risk Assessment===
===Traceability===

==Practical Basics of qNMR Analysis==

===Sample Preparation===
====Sample Weight====
====Solvents====

===qNMR Analysis===
There are several experimental NMR parameters that affect the quantitative determination concerning accuracy and precision of the NMR measurement. In order for the NMR result to be interpreted in a quantitative manner, these parameters must be be optimized.
====Excitation Pulse====
A 90° pulse angle or lower can be used for recording NMR spectra. Normally, a 90° pulse provides maximum intensity, but a smaller pulse angle such as 30° reduces the total experimental time.<ref>http://resources.mestrelab.com/qnmr-the-best-pulse-length-for-acquisition/</ref>
====Repetition Time====
The repetition time (<math>T_R</math>) is defined as the total time which is spent to acquire one scan of the NMR spectrum with fully relaxed signals. <math>T_R</math> is the sum of the acquisition time (<math>aq</math>) and the relaxation delay (<math>d_1</math>):

<math>T_R = aq + d_1</math>

<math>aq</math> is defined as the length of time during which the NMR signal is sampled and digitized and <math>d_1</math> is defined as the relaxation delay between repeated scans. Truncations of the FID, which can be detected as “wiggles” in the spectrum after Fourier transformation, should be avoided. The FID should be allowed to fully decay prior to the end of the acquisition period. <math>d_1</math> is defined as the relaxation delay between repeated scans. For the acquisition of quantitative NMR spectra, <math>T_R</math> should be chosen such that <math>T_R > 5 * T_1</math>, where <math>T_1</math> is the longitudinal relaxation time).

Section references: <references />
===Spectrum Evaluation===
====Phase Correction====
====Baseline Correction====
====Integration====

===Quantification Techniques===
====Internal Standard====
====External Standard====
====Standard Addition====
====PULCON====
====ERETIC====
====Tube in Tube====

===[[1H High-Precision Quantification|1H High-Precision Quantification]]===

===Validation===

====Nomenclature====
====Validation Process====
====GLP, GMP====
====System Suitability Test====

===Proficiency Tests===

===Quantitative Heteronuclear Analysis===
====15N qNMR====
====19F qNMR====
====31P qNMR====

===Education and Training===

===Applications===

====Food Screener====
====Lipid Analysis====
====Pharmaceuticals====
====Natural Materials====
====Polymer in Solutions====

If you have any questions and/or concerns, please contact [[User:Elina|Elina Zailer]] [mailto:zailer@spectralservice.de Email].

Quantitative NMR

2018-03-03T12:28:59Z

BryanHanson: /* Relative Quantification */

==Regulations==

===EU===

*[https://www.edqm.eu/en/european-pharmacopoeia-9th-edition European Pharmacopeia]
*[[German Arzneimittelbuch|Arzneimittelbuch]]

===US===
*[[US Pharmacopoeia|US Pharmacopoeia]]
===Japan===
*[[Japanese Pharmacopoeia| Japanese Pharmacopoeia]]

==Theoretical Basics of qNMR Analysis==

===Quantification===
The fundamental principle of qNMR is the direct proportionality of the signal intensity <math>I_X</math> in the NMR spectrum to the number of nuclei <math>N_X</math> giving rise to the signal:<math display="block">I_X = K_s \times N_X</math>where <math>K_s</math> is a spectrometer constant that is equal for all resonances in an NMR spectrum. The spectrometer constant <math>K_s</math> is affected by the repetition time, pulse excitation and broadband decoupling.

====Relative Quantification====
In the relative quantification method, the molar ratio between compounds <math>X</math> and <math>Y</math> is calculated by the following equation:

<math>
n_X / n_Y

= I_X / I_Y \times N_Y / N_X </math>

The spectrometer constant, <math>K_s</math> , influences all resonances in one spectrum equally, so the <math>K_s</math> term does not appear in the quantitative calculation.

====Absolute Quantification====
To calculate the absolute content of the main product, by-products, or impurities in a given sample, an internal standard of known concentration is used. Here, the contents are directly calculated from the NMR spectrum by the following equation:

<math>P_X = I_X / I_{IS} \centerdot N_{IS} / N_X \centerdot M_X / M_{IS} \centerdot m_{IS} /m_X \centerdot P_{IS}</math>

where <math>I</math> is the integral value, <math>N</math> represents the number of nuclei giving rise to the signal, <math>M</math> is the molar mass, <math>m</math> is the gravimetric weight and <math>P</math> is the purity of the analyte (<math>X</math> ) and the internal standard (<math>IS</math> ), respectively.

===Reference Standards===
===Parameter and Risk Assessment===
===Traceability===

==Practical Basics of qNMR Analysis==

===Sample Preparation===
====Sample Weight====
====Solvents====

===qNMR Analysis===
There are several experimental NMR parameters that affect the quantitative determination concerning accuracy and precision of the NMR measurement. In order for the NMR result to be interpreted in a quantitative manner, these parameters must be be optimized.
====Excitation Pulse====
A 90° pulse angle or lower can be used for recording NMR spectra. Normally, a 90° pulse provides maximum intensity, but a smaller pulse angle such as 30° reduces the total experimental time.<ref>http://resources.mestrelab.com/qnmr-the-best-pulse-length-for-acquisition/</ref>
====Repetition Time====
The repetition time (<math>T_R</math>) is defined as the total time which is spent to acquire one scan of the NMR spectrum with fully relaxed signals. <math>T_R</math> is the sum of the acquisition time (<math>aq</math>) and the relaxation delay (<math>d_1</math>):

<math>T_R = aq + d_1</math>

<math>aq</math> is defined as the length of time during which the NMR signal is sampled and digitized and <math>d_1</math> is defined as the relaxation delay between repeated scans. Truncations of the FID, which can be detected as “wiggles” in the spectrum after Fourier transformation, should be avoided. The FID should be allowed to fully decay prior to the end of the acquisition period. <math>d_1</math> is defined as the relaxation delay between repeated scans. For the acquisition of quantitative NMR spectra, <math>T_R</math> should be chosen such that <math>T_R > 5 * T_1</math>, where <math>T_1</math> is the longitudinal relaxation time).

Section references: <references />
===Spectrum Evaluation===
====Phase Correction====
====Baseline Correction====
====Integration====

===Quantification Techniques===
====Internal Standard====
====External Standard====
====Standard Addition====
====PULCON====
====ERETIC====
====Tube in Tube====

===[[1H High-Precision Quantification|1H High-Precision Quantification]]===

===Validation===

====Nomenclature====
====Validation Process====
====GLP, GMP====
====System Suitability Test====

===Proficiency Tests===

===Quantitative Heteronuclear Analysis===
====15N qNMR====
====19F qNMR====
====31P qNMR====

===Education and Training===

===Applications===

====Food Screener====
====Lipid Analysis====
====Pharmaceuticals====
====Natural Materials====
====Polymer in Solutions====

If you have any questions and/or concerns, please contact [[User:Elina|Elina Zailer]] [mailto:zailer@spectralservice.de Email].

Quantitative NMR

2018-03-03T12:24:11Z

BryanHanson: More mathml play

==Regulations==

===EU===

*[https://www.edqm.eu/en/european-pharmacopoeia-9th-edition European Pharmacopeia]
*[[German Arzneimittelbuch|Arzneimittelbuch]]

===US===
*[[US Pharmacopoeia|US Pharmacopoeia]]
===Japan===
*[[Japanese Pharmacopoeia| Japanese Pharmacopoeia]]

==Theoretical Basics of qNMR Analysis==

===Quantification===
The fundamental principle of qNMR is the direct proportionality of the signal intensity <math>I_X</math> in the NMR spectrum to the number of nuclei <math>N_X</math> giving rise to the signal:<math display="block">I_X = K_s \times N_X</math>where <math>K_s</math> is a spectrometer constant that is equal for all resonances in an NMR spectrum. The spectrometer constant <math>K_s</math> is affected by the repetition time, pulse excitation and broadband decoupling.

====Relative Quantification====
In the relative quantification method, the molar ratio between compounds <math>X</math> and <math>Y</math> is calculated by the following equation:

<math><mfrac><mrow> n_X </mrow> <mrow> n_Y </mrow></mfrac> = I_X / I_Y \times N_Y / N_X </math>

The spectrometer constant, <math>K_s</math> , influences all resonances in one spectrum equally, so the <math>K_s</math> term does not appear in the quantitative calculation.

====Absolute Quantification====
To calculate the absolute content of the main product, by-products, or impurities in a given sample, an internal standard of known concentration is used. Here, the contents are directly calculated from the NMR spectrum by the following equation:

<math>P_X = I_X / I_{IS} \centerdot N_{IS} / N_X \centerdot M_X / M_{IS} \centerdot m_{IS} /m_X \centerdot P_{IS}</math>

where <math>I</math> is the integral value, <math>N</math> represents the number of nuclei giving rise to the signal, <math>M</math> is the molar mass, <math>m</math> is the gravimetric weight and <math>P</math> is the purity of the analyte (<math>X</math> ) and the internal standard (<math>IS</math> ), respectively.

===Reference Standards===
===Parameter and Risk Assessment===
===Traceability===

==Practical Basics of qNMR Analysis==

===Sample Preparation===
====Sample Weight====
====Solvents====

===qNMR Analysis===
There are several experimental NMR parameters that affect the quantitative determination concerning accuracy and precision of the NMR measurement. In order for the NMR result to be interpreted in a quantitative manner, these parameters must be be optimized.
====Excitation Pulse====
A 90° pulse angle or lower can be used for recording NMR spectra. Normally, a 90° pulse provides maximum intensity, but a smaller pulse angle such as 30° reduces the total experimental time.<ref>http://resources.mestrelab.com/qnmr-the-best-pulse-length-for-acquisition/</ref>
====Repetition Time====
The repetition time (<math>T_R</math>) is defined as the total time which is spent to acquire one scan of the NMR spectrum with fully relaxed signals. <math>T_R</math> is the sum of the acquisition time (<math>aq</math>) and the relaxation delay (<math>d_1</math>):

<math>T_R = aq + d_1</math>

<math>aq</math> is defined as the length of time during which the NMR signal is sampled and digitized and <math>d_1</math> is defined as the relaxation delay between repeated scans. Truncations of the FID, which can be detected as “wiggles” in the spectrum after Fourier transformation, should be avoided. The FID should be allowed to fully decay prior to the end of the acquisition period. <math>d_1</math> is defined as the relaxation delay between repeated scans. For the acquisition of quantitative NMR spectra, <math>T_R</math> should be chosen such that <math>T_R > 5 * T_1</math>, where <math>T_1</math> is the longitudinal relaxation time).

Section references: <references />
===Spectrum Evaluation===
====Phase Correction====
====Baseline Correction====
====Integration====

===Quantification Techniques===
====Internal Standard====
====External Standard====
====Standard Addition====
====PULCON====
====ERETIC====
====Tube in Tube====

===[[1H High-Precision Quantification|1H High-Precision Quantification]]===

===Validation===

====Nomenclature====
====Validation Process====
====GLP, GMP====
====System Suitability Test====

===Proficiency Tests===

===Quantitative Heteronuclear Analysis===
====15N qNMR====
====19F qNMR====
====31P qNMR====

===Education and Training===

===Applications===

====Food Screener====
====Lipid Analysis====
====Pharmaceuticals====
====Natural Materials====
====Polymer in Solutions====

If you have any questions and/or concerns, please contact [[User:Elina|Elina Zailer]] [mailto:zailer@spectralservice.de Email].

Quantitative NMR

2018-03-03T12:20:46Z

BryanHanson: Adjusting mathml formating (test)

==Regulations==

===EU===

*[https://www.edqm.eu/en/european-pharmacopoeia-9th-edition European Pharmacopeia]
*[[German Arzneimittelbuch|Arzneimittelbuch]]

===US===
*[[US Pharmacopoeia|US Pharmacopoeia]]
===Japan===
*[[Japanese Pharmacopoeia| Japanese Pharmacopoeia]]

==Theoretical Basics of qNMR Analysis==

===Quantification===
The fundamental principle of qNMR is the direct proportionality of the signal intensity <math>I_X</math> in the NMR spectrum to the number of nuclei <math>N_X</math> giving rise to the signal:<math display="block">I_X = K_s \times N_X</math>where <math>K_s</math> is a spectrometer constant that is equal for all resonances in an NMR spectrum. The spectrometer constant <math>K_s</math> is affected by the repetition time, pulse excitation and broadband decoupling.

====Relative Quantification====
In the relative quantification method, the molar ratio between compounds <math>X</math> and <math>Y</math> is calculated by the following equation:

<math>n_X / n_Y = I_X / I_Y \centerdot N_Y / N_X </math>

The spectrometer constant, <math>K_s</math> , influences all resonances in one spectrum equally, so the <math>K_s</math> term does not appear in the quantitative calculation.

====Absolute Quantification====
To calculate the absolute content of the main product, by-products, or impurities in a given sample, an internal standard of known concentration is used. Here, the contents are directly calculated from the NMR spectrum by the following equation:

<math>P_X = I_X / I_{IS} \centerdot N_{IS} / N_X \centerdot M_X / M_{IS} \centerdot m_{IS} /m_X \centerdot P_{IS}</math>

where <math>I</math> is the integral value, <math>N</math> represents the number of nuclei giving rise to the signal, <math>M</math> is the molar mass, <math>m</math> is the gravimetric weight and <math>P</math> is the purity of the analyte (<math>X</math> ) and the internal standard (<math>IS</math> ), respectively.

===Reference Standards===
===Parameter and Risk Assessment===
===Traceability===

==Practical Basics of qNMR Analysis==

===Sample Preparation===
====Sample Weight====
====Solvents====

===qNMR Analysis===
There are several experimental NMR parameters that affect the quantitative determination concerning accuracy and precision of the NMR measurement. In order for the NMR result to be interpreted in a quantitative manner, these parameters must be be optimized.
====Excitation Pulse====
A 90° pulse angle or lower can be used for recording NMR spectra. Normally, a 90° pulse provides maximum intensity, but a smaller pulse angle such as 30° reduces the total experimental time.<ref>http://resources.mestrelab.com/qnmr-the-best-pulse-length-for-acquisition/</ref>
====Repetition Time====
The repetition time (<math>T_R</math>) is defined as the total time which is spent to acquire one scan of the NMR spectrum with fully relaxed signals. <math>T_R</math> is the sum of the acquisition time (<math>aq</math>) and the relaxation delay (<math>d_1</math>):

<math>T_R = aq + d_1</math>

<math>aq</math> is defined as the length of time during which the NMR signal is sampled and digitized and <math>d_1</math> is defined as the relaxation delay between repeated scans. Truncations of the FID, which can be detected as “wiggles” in the spectrum after Fourier transformation, should be avoided. The FID should be allowed to fully decay prior to the end of the acquisition period. <math>d_1</math> is defined as the relaxation delay between repeated scans. For the acquisition of quantitative NMR spectra, <math>T_R</math> should be chosen such that <math>T_R > 5 * T_1</math>, where <math>T_1</math> is the longitudinal relaxation time).

Section references: <references />
===Spectrum Evaluation===
====Phase Correction====
====Baseline Correction====
====Integration====

===Quantification Techniques===
====Internal Standard====
====External Standard====
====Standard Addition====
====PULCON====
====ERETIC====
====Tube in Tube====

===[[1H High-Precision Quantification|1H High-Precision Quantification]]===

===Validation===

====Nomenclature====
====Validation Process====
====GLP, GMP====
====System Suitability Test====

===Proficiency Tests===

===Quantitative Heteronuclear Analysis===
====15N qNMR====
====19F qNMR====
====31P qNMR====

===Education and Training===

===Applications===

====Food Screener====
====Lipid Analysis====
====Pharmaceuticals====
====Natural Materials====
====Polymer in Solutions====

If you have any questions and/or concerns, please contact [[User:Elina|Elina Zailer]] [mailto:zailer@spectralservice.de Email].

Talk:1H High-Precision Quantification

2018-03-02T19:16:23Z

BryanHanson: Comment on an unclear sentence

The second sentence of the first paragraph is a bit mysterious: What are the "things" referred to? My sense is the meaning is "Many experimental aspects need to be carefully controlled in order to achieve this high level of precision."

1H High-Precision Quantification

2018-03-02T19:12:04Z

BryanHanson: Tiny language edits

High-precision (HP) measurement is in principle nothing more than normal [[Quantitative NMR|qNMR]], however every individual step has to be performed as precisely as possible. Some things have to be considered in addition since their effect is detectable in the lower uncertainty range for the measurement.
Usually, in [[Quantitative NMR|qNMR]], a measurement is considered as “high-precision” if the measurement’s uncertainty is distinctly below 0.5 %. High-precision qNMR is usually applied for the purity detection of almost pure substances (not mixtures).

==Sample Preparation==
An internal calibrant is needed to achieve this high precision. Since the uncertainty value of the calibrant’s purity is one of the main contributions to the combined uncertainty of the measurement, a very homogeneous and well-defined calibrant has to be chosen, ideally with an uncertainty level below 0.1 %.
The weighing of an analyte and internal calibrant has to be performed using an ultra-micro balance under the highest metrological safeguards; an anti-static kit<ref>http://www.mt.com/us/en/home/products/Laboratory_Weighing_Solutions/Accessories/anti_static_devices.html</ref> can help to prevent weighing errors due to static charges.
The air buoyancy effect can lead to a systematic error up to some tenths of parts per million when the calibrant and analyte have very different density values. This contribution can be subtracted out when the density of the material is known.
A very helpful Guidebook for the weighing procedure is provided by Mettler Toledo<ref>„Weighing the Right Way“, Mettler Toledo: http://us.mt.com/dam/MT-NA/WeighMatter/Weighing_the_Right_Way_Brochure.pdf</ref>.

==Data Acquisition==
===Relaxation===
The time between two pulses (recycle delay) should be long enough to enable (almost) full relaxation. The sum of the relaxation delay and the acquisition time should exceed seven times the highest T1 value of all evaluated signals. Theoretically, the factor seven will result in up to 99.9 % of the maximum signal intensity being measured.
===Pulse Angle===
The employment of 90-degree observation pulse leads to the maximum signal intensity with respect to unit measurement time.<ref>http://resources.mestrelab.com/qnmr-the-best-pulse-length-for-acquisition/</ref>

===Transmitter Offset===
Pulses used for hard excitation can never be considered as truly uniform over the entire spectral range. The excitation decreases with increasing distance from the transmitter offset. This effect can actually be neglected for “normal” qNMR measurements in the 1 % uncertainty level, but its effect can be observed at the high-precision level.
This phenomenon can lead to systematic errors of some ppm if distances of the signals relative (analyte and calibrant) to the transmitter offset are distinctly different. To minimize this effect the transmitter offset should be set in the middle of the signals that have to be evaluated.
===Digital Filter===
Digital filters, such as “baseopt” for Bruker instruments, should be used to minimize baseline problems directly at the data acquisition level.
==Data Processing==
===Phase Correction===
Automatic phase corrections are generally not precise enough for HP-qNMR. The phase should be corrected manually by careful visual control.
===Baseline Correction===
The baseline must be corrected very carefully. A number of suitable algorithms are available for that purpose, such as “polynomial fit”. It is necessary to pay attention that only the baseline itself, but not the actual signal is corrected.

[[File:HPqNMR-BaselineCorrection.png|400px|border|Bad example for baseline correction]]

===Partial Baseline Correction===
It is sometimes necessary to perform an additional, "local" partial baseline correction in the region of a signal which has to be evaluated, e.g. when they are superimposed with very broad exchangeable Hs.
The correction of the integral’s bias and slope should not be used, and will not be a factor if the baseline was corrected properly.
Applicable tools are, for example, the partial polynomial fit correction (e.g. “absf” for Bruker), or manual multipoint baseline correction.
[[File:HPqNMR-Multipont.PNG|500px|border|Example for multipoint baseline correction. Corrected spectrum is on the top.]]

==Spectrum Evaluation==
===Integral Limits===
Since it is practically impossible to determine the integral of the entire line, the main goal of setting the integration limit is to always evaluate the same amount of the signal.
The shape of the NMR signal is usually described by the Lorentz Function. Using an integration setting of 64 times the half-height width achieves a 99% capture of the main signal (ignoring the 13C satellites), whereas a setting of 630 times the half-height width was required to provide a 99.9% gain of main signal intensity [1]. Therefore, a constant factor (x –times half-height width) should be selected; the integral width is dependant on the linewidth.
A pragmatic compromise is to select for measurement only quite narrow signals for non-labile protons, and set the integral limits uniquely outside the 13C satellites. This only causes small errors which are acceptable even for high-precision measurement.
<ref>http://resources.mestrelab.com/getting-really-fussy-about-integration</ref>

[[File:HPqNMR-IntRegion.PNG|300px|border|Integration region ]]

===Signal to Noise Ratio (SN)===
The SN has the main and direct influence on the measurement accuracy. For example, with an SN of 100, only an uncertainty level of ~100 can be achieved. SN of more than 10,000 is necessary to reach the uncertainty level of 0.1 %.
An experimental equation for this dependency is described by Hays et al.<ref>Hays, P.; Schoenberger, T. Anal. Bioanal. Chem. 2014, 406, 7397−7400.</ref>

===Impurity Signals===
Integration regions should be preferably totally free of impurity signals - even in the per mill area level - in comparison to the main signal; usually, this is not the case in practice. So the integrals of the impurity signal have to be subtracted from the main signal.
For example, MestReNova's “Edited Sum” function is ideally suited for this purpose. It is a hybrid of integration (for the overall signal) and line fitting (for the determination of the impurity’s amount which has to be subtracted). The uncertainty is mainly influenced by the much more precise integration, whereas the uncertainty contribution of the less precise line fitting can be almost neglected for small impurity amounts.<ref>Schoenberger, T.; Menges, S.; Bernstein M.A.; Pérez, M.; Seoane, F.; Sýkora, S.; Cobas, C.: Anal. Chem. 2016, 88(7), 3836-3843.</ref>

[[File:HPqNMR-EdSum.PNG|border|500px|MNova's Edited Sum function]]
<references />

Chemometrics

2018-03-02T18:08:59Z

BryanHanson: Misc language edits; still can't get images uploaded.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). An alternative is the clupa method, which is quite robust (Figure 4 coming soon).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref> [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group should contain as many samples as possible (not less than 20 are recommended). The total number of samples in a calibration set should be at least 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to principles of experimental design.

==== Multivariate Models for Prediction ====

In multivariate modeling, the data set would be composed of a set of spectra along with several response variables measured for each of the samples. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be used. Partial least squares regression (PLSR) is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of latent variables (LVs), and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors. These, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

If outliers are identified, the model has to be recalculated. Outliers also have to be excluded from the validation test set.

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

==== Cross-Validation ====
The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2018-03-02T17:22:39Z

BryanHanson: Added reference to clupa alignment method.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). An alternative is the clupa method, which is quite robust (Figure 4).<ref>Vu TN, Valkenborg D, Smets K, Verwaest KA, Dommisse R, Lemiere F, Verschoren A, Goethals B, Laukens K. An integrated workflow for robust alignment and simplified quantitative analysis of NMR spectrometry data" ''BMC Bioinformatics'' vol. 12 pg. 405 (2011).</ref> [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-21T14:50:17Z

BryanHanson: Added link to nomenclature wiki

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS. In addition to the material here, you may find the [[Nomenclature|NMR Nomenclature]] wiki helpful.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-20T18:06:45Z

BryanHanson: Removed one of the public data set links which was not really about data, was in the process of moving, and is linked to elsewhere on the wiki

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-20T18:01:44Z

BryanHanson: One more stray character

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 
====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-20T17:58:50Z

BryanHanson: Removed some stray characters & further clean up of software list

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] [2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] [3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] [2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] [2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] [2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] [3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] [4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] [5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] [6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] [7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] [8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] See also [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]

[https://www.r-project.org/ R computing system] See also CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)

[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] [https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] [http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] [http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] [http://www.cs.umass.edu/~elm/ICA/ RADICAL] [http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-20T17:53:07Z

BryanHanson: Cleaned up software links

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB]
* [http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm Multi-block toolbox for Matlab]
[https://www.r-project.org/ R computing system]
* CRAN task view on [https://cran.r-project.org/web/views/ChemPhys.html chemometrics and physics] (an overview of packages)
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] [http://www.chimiometrie.fr/saisir_webpage.html SAISIR] [http://umetrics.com/products/simca SIMCA-P] [https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] [https://infometrix.com/pirouette/ Pirouette] [http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-12-20T17:43:28Z

BryanHanson: Added a bit to HCA section; still needs elaboration.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set. When conducting HCA, there are two main decisions that must be made. One is how to calculate distances between the sample spectra. The second is how to choose the clusters (called aglomeration). Most chemometric software provides several options for each of these processes.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-10-19T17:31:01Z

BryanHanson: Further explanation re: S^T notation.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra. This practice seems to arise from an infrequently used graphical representation of the matrix multiplication process needed to solve the system of equations, in which it was visually useful to imagine the spectra in columns rather than rows.</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-10-06T19:04:34Z

BryanHanson: /* Connection between PCA, ICA and MCR */ Added a note about alternative matrix notations.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E''' (or in some literature '''D''' = '''CST''' + '''E''')<ref>When the notation '''ST''' is used its dimensions must be the same as '''S''' in order for the matrices to be conformable. In this case the original '''S''' before transposition has dimensions ''p'' chemical shifts x ''m'' unique/pure spectra</ref>

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-09-22T14:04:05Z

BryanHanson: Missed bolding a symbol

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E'''

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix '''E''' is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra in the case of MCR) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-09-22T13:55:17Z

BryanHanson: Added section on mathematical connections between ICA, PCA and MCR.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose the same bilinear model on the data. This model can be written in matrix notation as:

'''D''' = '''CS''' + '''E'''

where '''D''' is a matrix containing the spectra (''n'' samples x ''p'' chemical shifts). The spectra are presumed to be mixtures of pure spectra. Matrix '''C''' contains the concentrations/weights (''n'' samples x ''m'' unique/pure spectra). Matrix '''S''' contains pure spectra (''m'' unique/pure spectra x ''p'' chemical shifts). Matrix E is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied. In addition, the choice of ''m'', the number of unique components (PCs, ICs or pure spectra) is an important issue separate from the method used to decompose matrix '''D'''.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].

Chemometrics

2017-09-22T13:44:04Z

BryanHanson: Updating ICA and MCR sections. Adding a section on the math.

==Background==
The primary aims of chemometrics are to separate useful information from noise and to find the crucial patterns in the experimental data. The central idea is to reduce the dimensionality of the data consisting of a large number of measured variables (in this case, NMR shifts) while retaining as much useful information as possible. There are typically two phases in chemometrics: 1. ''Exploratory data analysis'' in which one seeks to get an overall sense of the data set. This is often, but not always followed by 2) ''Supervised modeling'' aimed at classification of new samples or building a multivariate regression model with predictive value.

This wiki is not intended as a comprehensive text book (many fine ones exist, see [[#General works on chemometrics|General works on chemometrics]]), but rather as a summary of major concepts and techniques, with pointers to key literature and resources. While the focus here is on chemometrics methods for use with NMR, much of the background presented here applies to other types of spectroscopy as well, such as IR, Raman and MS.

==Typical Steps in a Chemometrics Analysis of NMR Data==
Figure 1 shows a typical workflow for chemometric analysis of NMR data. It includes steps such as sample preparation, spectra acquisition, pre-processing, exploratory analysis, as well as multivariate modeling (including validation and optimization). Each of these steps is discussed below. [[File:Fig1_YM_FlowChart.png|thumb]]

=== At the Spectrometer ===
''Sample preparation.'' For high-resolution NMR of liquids (beverages, clinical samples) the samples are generally prepared by adding proper solvent (frequently containing buffer) and internal standard. For samples with pH-dependent compounds, pH control is required. For chemometric analysis standard procedures should be followed to ensure reproducibility and comparability when preparing a series of samples. Karakach, Wentzell and Walter discuss typical sources of error encountered in metabolomic studies which apply to most chemometric workflows.<ref>Karakach, T. K., Wentzell, P. D., Walter, J. A. Characterization of the measurement error structure in 1D H-1 NMR data for metabolomics studies ''Analytica Chimica Acta'' '''2009''' 636: 163-174.</ref>

''Spectra acquisition.'' Spectra acquisition is one of the most challenging steps in the optimization of NMR analysis. For chemometrics purposes it should be assured that exactly the same pulse program and acquisition parameters are used for ''all'' samples. It should be also verified that any solvent suppression scheme does not affect (or affects equally) the whole series of signals located close to the suppressed region.

''Phasing, baseline correction and referencing.'' Adequate baseline and phase correction is fundamental for chemometric analysis. The phase and the baseline of the spectra is usually corrected manually or with one of several automatic correction functions from the spectrometer software (Figure 2). At this stage one should also reference each spectrum to an internal reference peak (TSP or TMS).

Generally speaking, the steps above should follow the recommendations given in the [[Quantitative NMR]] wiki page.[[File:Fig2_YM_BCHMF.png|thumb|Figure 2. Baseline correction by the moving minimum algorithm for 1H NMR signal of hydroxymethylfurfural in cola sample]]

===Pre-Processing===
Most or all of the following steps should be considered once the spectra have been exported from the spectrometer and prior to either unsupervised or supervised modeling.

''Preliminary inspection.'' Generally, it is desirable to plot the data at this stage to look for overtly odd spectra/samples. This may give the first indications of outliers that may need to be removed once they are verified by more formal methods.

''Peak alignment.'' A serious consideration, especially for aqueous samples, is the chemical shift variation that may occur from sample-to-sample or even from peak-to-peak. Overall sample-to-sample variations are due to small variations in spectrometer frequency and are generally addressed by referencing to the internal standard. The peak-to-peak chemical shift variations due to differences in, for example, pH or concentrations, are much more difficult to handle. One method for addressing this chemical shift variability across spectra is by means of the so-called bucketing (also known as binning). The procedure consists of segmenting a spectrum into small areas (buckets or bins) and taking the area under the spectrum for each segment. The bucket width typically varies between 0.01 and 0.05 ppm for 1H NMR. Other algorithms are available for binning which take into account the structure of peaks and try to avoid arbitrarily separating peaks that belong together, that is, the parts of a multiplet.<ref>De Meyer, T., Sinnaeve, D., Van Gasse, B., Tsiporkova, E., Rietzschel, E. R., De Buyzere, M. L., Gillebert, T. C., Bekaert, S., Martins, J. C., Van Criekinge, W. NMR-based characterization of metabolic alterations in hypertension using an adaptive, intelligent binning algorithm ''Analytical Chemistry'' '''2008''' 80: 3783-3790.</ref> At one time, binning was desirable as a data reduction method since there are fewer data points in each spectrum after binning. However, algorithms and computers are now more than fast enough to handle un-binned spectra, so binning for the sake of speed is no longer done.

The major drawback of this procedure is the loss of a considerable amount of information in the original spectra. There are several good alternatives to binning which involve some form of peak alignment without data reduction. For example, the interval correlation optimized shifting algorithm (e.g., icoshift) <ref>Savorani, F., Tomasi, G., Engelsen, S. B. Icoshift: A versatile tool for the rapid alignment of 1D NMR spectra. ''J. Magn. Reson.'' '''2010''', 202: 190-202. https://doi.org/10.1016/j.jmr.2009.11.012</ref> splits a spectral data set into intervals and co-shifts each vector left-right to get the maximum correlation towards a target spectrum (Figure 3). [[File:Fig3_YM_icoshift.png|thumb|Figure 3. Alignment by the icoshift algorithm for 13C NMR signals in the carboxyl group region for a set of milk samples]]

''Normalization.'' Many NMR samples will consist of a biofluid or extract of some kind diluted by an appropriate solvent. In these cases it is often necessary to normalize the individual spectra in some manner in order to compensate for concentration differences. There are a number of normalization algorithms available.<ref>Craig, A., Cloarec, O., Holmes, E., Nicholson, J. K., Lindon, J. C. Scaling and normalization effects in NMR spectroscopic metabonomic data sets ''Analytical Chemistry'' '''2006''' 78: 2262-2267.</ref><ref>Romano, R., Santini, M. T., Indovina, P. L. A time-domain algorithm for NMR spectral normalization ''Journal of Magnetic Resonance'' '''2000''' 146: 89-99.</ref><ref>van den Berg, R. A., Hoefsloot, H. C. J., Westerhuis, J. A., Smilde, A. K., van der Werf, M. J. Centering, scaling, and transformations: improving the biological information content of metabolomics data ''BMC Genomics'' '''2006''' 7: 15.</ref><ref>Zhang, S. C., Zheng, C., Lanza, I. R., Nair, K. S., Raftery, D., Vitek, O. Interdependence of Signal Processing and Analysis of Urine H-1 NMR Spectra for Metabolic Profiling ''Analytical Chemistry'' '''2009''' 81: 6080-6088.</ref>

''Data cleaning.'' To compact the data and speed up the analysis, it is desirable to remove spectral regions containing only solvent peaks and internal standards. If there are large regions consisting only of noise, these may be removed as well.

''Baseline correction.'' It may be desirable to do further baseline correction at this point.

===Exploratory Chemometrics for 1D NMR Data (Unsupervised Modeling)===
==== Data Reduction ====
Most chemometric methods are based on the idea of latent variables (LVs). Initial (measured) variables can be combined and described by a smaller number of LVs, which describe the underlying structure of the data. This is referred to as data reduction. Just how this is done and the interpretation of the results depends upon the technique. In addition to simplifying the data set, which can improve interpretibility, the smaller data set requires less computational time.

Data reduction works because NMR data is highly redundant. For instance, a single peak may be composed of a dozen data points. If this peak represents a particular compound across many samples, each of the dozen data points rises and falls in the same way as the concentration of that compound changes. Thus, any one of the dozen data points can stand in for all of them. Further, noise is random and data reduction techniques can identify variables (particular chemical shifts) that are varying randomly and not contributing to any overall trend. Conceptually then, data reduction methods return new (abstract) variables that describe the original data set using many fewer variables.

All data reduction methods generally require mean-centering and scaling. The mean-centered matrix is obtained by subtracting the mean intensity for each of the variables from each spectrum. Scaling methods are data pre-treatment approaches that divide each variable by a scaling factor, which is different for each variable. For example, dividing each variable by it's standard deviation sets all peaks, including noise, to the same scale and thus small peaks contribute equally to larger peaks. In contrast, not scaling at all means larger peaks have more influence on the outcome of the analysis. Pareto scaling scales by the square root of the standard deviation and thus balances the effects of large and small peaks.

===== PCA =====
The oldest and most common LV projection method is principal component analysis (PCA). PCA is based on transformation of the original data into a new set of a few orthogonal LVs (known as principal components, PCs), which describe most of the variation in the data. The first PC accounts for the maximal variation of data, while each successive PC does not correlate (or interact) with the previous PCs and expresses as much of the remaining variation as possible. Three quantities result from a PCA exploration:
* The amount of variance explained by each PC. These are typically displayed in a scree plot, and used to determine how many PCs are necessary to describe the data. For NMR data there are typically thousands of data points in a spectrum, but in the end they may be represented by only 3-10 PCs.
* Scores. After PCA each sample resides in a reduced-dimension, new "space" in which each sample is characterized by scores associated with each PC. Score plots characterize the samples in this new space, and often show clustering or trends, and can be used to identify outliers.
* Loadings. The loadings represent the importance or contribution of each variable (chemical shift) to the scores. Large loadings indicate chemical shifts (and thus, compounds) that contribute to the separation of the samples in the score plot.
The chief drawback of PCA is that the results (scores and loadings) exist in an abstract space, and as such, can be difficult to interpret. For example, there is no guarantee that a loading vector represents anything real, such as the spectrum of a particular compound. Instead, a loading vector will generally be composed of some combination of features from several components. So while a PCA score plot may give a satisfactory separation of samples into groups, for instance treatment and control, interpretation of the loadings is generally limited. ICA, discussed below, is one way to overcome these limitations.

===== AOV-PCA =====
Standard PCA is a technique for finding clustering among the samples, but it does not use information about which samples belong to which group. AOV-PCA is a method that conducts PCA with the group membership in mind, and thus blends in concepts of analysis of variance.<ref>Pinto, Rui Climaco, Bosc, Veronique, Nocairi, H., Barros, Antonio S., Rutledge, Douglas N. Using ANOVA-PCA for discriminant analysis: Application to the study of mid-infrared spectra of carraghenan gels as a function of concentration and temperature ''Analytica Chimica Acta'' '''2008''' 629: 47-55 http://dx.doi.org/10.1016/j.aca.2008.09.02</ref> <ref>Harrington, PD, Vieira, NE, Espinoza, J, Nien, JK, Romero, R, Yergey, AL Analysis of variance-principal component analysis: A soft tool for proteomic discovery ''Analytica Chimica Acta'' '''2005''' 544: 118-127 http://dx.doi.org/10.1016/j.aca.2005.02.042</ref> <ref>D. Jouan-Rimbald Bouveresse, R. Climaco Pinto, L. M. Schmidtke, N. Locquet, D. N. Rutledge Identification of significant factors by an extension of ANOVA-PCA based on multi-block analysis ''Chemometrics and Intelligent Laboratory Systems'' '''2011''' 106: 173-182.</ref>

===== ICA =====
Independent component analysis (ICA) is technique which is similar to PCA, but rather than maximizing the explained variation as PCA does, ICA attempts to find pure signals (independent components, ICs) which are statistically independent from each other. In chemical terms, an IC is a spectrum of a pure compound, and thus interpretation is straightfoward. ICA was originally proposed for the blind-source-separation problem in signal processing. The results of an ICA exploration are:
* A set of pure spectra representing the various components in the data set ("signal").
* A matrix of weights reflecting how much of each pure spectrum is present in each sample.
It should be kept in mind that in ICA the pure spectra/compounds that are identified must be independent from each other. If two compounds in a mixture are in equilibrium with each other, they are not independent, and the extract spectrum will have features from each compound.

===== MCR =====
Multivariate curve resolution-alternating least squares (MCR-ALS) method is closely related to both PCA and ICA. MCR is mathematically similar to PCA in that it seeks solutions that account for the most variation possible. However, MCR adds constraints that the solutions (the extracted spectra) must be non-negative, and the concentrations (weights) must sum to one. Additional constraints can also be applied. MCR is an algorithm that can extract analyte profiles in the presence of interferences even in the case of a high degree of overlap. This means that, contrary to partial least squares regression (PLSR, see below), quantification of samples containing unexpected components, not represented in the calibration mixtures, is possible.

===== Connection between PCA, ICA and MCR =====
The methods discussed above all impose a bilinear model on the data. This model can be written as

where '''D''' is a matrix containing the spectra (samples in rows). The spectra are presumed to be mixtures of pure spectra. Matrix C contains a certain number of pure spectra and matrix S contains the concentrations/weights. Matrix E is an error term. The difference between each of these methods are the algorithms used to optimize the solution and any constraints that are applied.

is one of the advanced chemometric methods used for the exploration of NMR data. MCR-ALS mathematically decomposes a global mixed instrumental response matrix '''D''' containing raw mixture signals into two data matrices '''C''' and '''ST''', representing spectra of presumably pure components and their contributions (weights) to the original data, respectively. Singular value decomposition (SVD), the first step in MCR-ALS optimization, reduces the data and determines the rank (number of significant components) of the data set. After that, the ALS algorithm iteratively searches for the matrices '''C''' and '''ST''' that best fit the reduced data. Additional knowledge can be used to reduce improve the results. Typically, one constrains the results so that the fitted spectra are all positive, and the concentrations must be positive and sum to one.

==== HCA ====
Hierarchical cluster analysis (HCA) is a type of chemometric analysis which analyzes similarities between samples. HCA can be performed on the original spectra, or on the PCA scores of a reduced data set.

===Exploratory Chemometrics for 2D NMR Data===
Several chemometric methods are available for modeling of 2D NMR signals (such as DOSY, HSQC, HMBC, TOCSY, etc). Parallel factor analysis (PARAFAC) is the most advanced method for the investigation of high-dimensional data. PARAFAC is a generalization of PCA for higher order arrays, and uses an ALS algorithm to iteratively find mixture components. Each of them consists of three informative directions, namely, one score vector and two loadings vectors, as well as a matrix of error. Like in MCR-ALS, constraining the PARAFAC solutions by means of orthogonality and non-negativity criteria can be helpful in terms of interpretability and stability of the solution.

Loadings resolved using PARAFAC methods are not hampered with the rotational ambiguity, and are, therefore, directly interpretable. The results of the PARAFAC model are assessed using model fit and corcondia (core consistency), where the latter should be equal to 100% in the ideal case.

The improved PARAFAC algorithm, PARAFAC2, decomposes higher data arrays into loading matrices, but does not impose such strong restrictions on the data structure. PARAFAC2 does not assume that the shape (or even length) of the elution profile of an analyte is the same in each sample; therefore, it is more sensitive to noise.

Tucker3 is another generalization of PCA for higher order data. Elements of the experimental 3D matrix are decomposed into three loading matrices and a core tensor. Like PCA, Tucker3 has rotational freedom and any model can be rotated without changing the precision of the description of experimental data. Unlike in PARAFAC analysis, the core tensor, '''G''', is not superdiagonal and allows analysis of interaction between different modes.

=== Supervised Modeling with 1D NMR Data ===
All available supervised chemometric methods can be used for analysis of NMR spectra. Generally they are aimed at tackling two main objectives: classification of objects or the development of multivariate models such as regression analysis.

==== Classification of Objects ====
For classification purposes, each predefined group has to contain as much samples as possible (not less than 20 are recommended). The number of samples in a calibration set has not to be less than 50 for multivariate calibration. Collinearities of variables caused by correlated concentrations in calibration samples have to be avoided. Therefore, the composition of calibration mixtures should be chosen according to experimental design.

==== Multivariate Models for Prediction ====

The multivariate model has to be recalculated without the detected outliers. Outliers also have to be excluded from the validation test set. The samples used to construct a multivariate model and for its validation have to be authentic and the desired parameter for classification has to be verified (e.g., by ''a priori'' knowledge obtained during sampling or by application of an adequate reference method).

After data reduction, further treatment by variable selection methods is recommended. Variable selection is the process of selecting a subset of only relevant features for use in model construction. For variable selection of NMR data clustering of latent variables (CLV) is recommended <ref>Vigneau, E., Qannari, E. M. Clustering of variables around latent components. ''Commun. Stat. Simul. C.'' '''2003''', 32: 1131–1150. http://dx.doi.org/10.1081/SAC-120023882</ref>. Significant variables are selected using a calibration data set where the classes are known in advance.

For quantitative determination of multiple analytes with spectral overlap, multivariate regression methods can be also used. PLSR is considered a standard method of this category. PLSR is a linear regression based method for relating a set of collinear and noisy predictor variables, '''X''' (for example, spectra profiles), with one or more response variables, '''Y''' (analyte concentrations). PLSR reduces data dimensionality by calculating a set of LVs, and each of them is checked for predictive power. In PLSR, both '''X''' and '''Y''' matrices are decomposed like in PCA providing the score and loading vectors, which, however, differ from those provided by PCA, and describe a relation between '''X''' and '''Y'''. The number of significant PLS factors in a calibration model can be determined using cross-validation. The most time-consuming part of the PLSR approach is the preparation of a representative calibration set with known concentration of analytes.

==== Cross-Validation ====
It is thus essential that the validation is performed adequately and produces reliable results. The validation of a multivariate model can be performed by cross-validation or test set validation.

One round of cross-validation involves partitioning a data set into complementary subsets, performing the analysis on one subset (called the training set), and validating the analysis on the other subset (called the validation set or testing set). To reduce variability, multiple rounds of cross-validation are performed using different partitions, and the validation results are averaged over the rounds.

Because cross-validation is optimistic, it should be combined with other techniques such as test set validation. Test set validation should be used if there are enough samples in the data table, for instance more than 50. A test set should contain 20-40% of the full data table. The calibration and test sets should cover the whole sample population.

=== Analytical Method Validation ===
Parameters that have to be validated for the specific purpose are summarized in the [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Eurolab Guidelines, Part II].

==References==
<references />
==Resources==
===Selected Literature===
====General works on chemometrics====
[1] Varmuza K., Filzmoser P. ''Introduction to Multivariate Statistical Analysis in Chemometrics'' CRC Press (2009).

[2] Otto, M. ''Chemometrics: Statistics and Computer Application in Analytical Chemistry'' 3rd Edition Wiley-VCH (2017).

[3] Wehrens, R. ''Chemometrics with R: Multivariate Data Analysis in the Natural Sciences and Life Sciences'' Springer (2011). 
====Principal component analysis (PCA)====
[1] [https://www.researchgate.net/profile/Kim_Esbensen/publication/222347483_Principal_Component_Analysis/links/00b4952c66e796fc2d000000.pdf Wold, S., Esbensen, K., Geladi, P. Principal component analysis. ''Chemometr. Intell. Lab.'' '''1987''', 2: 37-52.] 

====Multivariate curve resolution (MCR) ====
[1] [http://dx.doi.org/10.1080/10408340600970005 de Juan, A., Tauler, R. Multivariate curve resolution (MCR) from 2000: Progress in concepts and applications. ''Crit. Rev. Anal. Chem.'' '''2006''', 36: 163-176.] 
[2] [https://www.researchgate.net/publication/223888647_Chemometrics_applied_to_unravel_multicomponent_processes_and_mixtures_Revisiting_latest_trends_in_multivarate_resolution de Juan, A., Tauler, R. Chemometrics applied to unravel multicomponent processes and mixtures. Revisiting latest trends in multivariate resolution. ''Anal. Chim. Acta.'' '''2003''', 500: 195-210.] 
[3] [https://doi.org/10.1016/j.aca.2012.12.028 Ruckebusch, C., Blanchet, L. Multivariate curve resolution: A review of advanced and tailored applications and challenges. ''Anal. Chim. Acta.'' '''2013''', 765: 28-36.] 
====Independent component analysis (ICA)====
[1] [https://doi.org/10.1016/j.trac.2013.03.013 Rutledge, D. N., Jouan-Rimbaud Bouveresse, D. Independent Components Analysis with the JADE algorithm. ''TrAC-Trend. Anal. Chem.'' '''2013''', 50: 22-32.] 
[2] [https://www.cs.helsinki.fi/u/ahyvarin/papers/bookfinal_ICA.pdf Hyvaerinen, A., Karhunen, J., Oja, E. Independent Component Analysis, Wiley: New York, 2001, 475 p.] 
====PARAFAC and TUCKER====
[1] [https://www.researchgate.net/publication/223735298_PARAFAC_tutorial_and_applications_Chemom_Intell_Lab_Syst Bro, R. PARAFAC Tutorial and applications. ''Chemometr. Intell. Lab.'' '''1997''', 38: 149-171.] [2] Tucker, L. R. The extension of factor analysis to three-dimensional matrices. In ''Contributions to mathematical psychology''; Fredericksen, N., Gulliksen, H., Eds.; Holt, Rinehart & Winston: New York, 1964. [3] Keirs, H. A. L. Towards a standardized notation and terminology in multiway analysis. ''Journal of Chemometrics'' '''2000''' 14: 105-122.

[4] Smilde, A. Bro, R. Geladi, R. Multiway Analysis: Applications in the Chemical Sciences. Wiley: 2004, 381 p.
====Multivariate regression models====
[1] [http://www.ece.mcmaster.ca/faculty/reilly/ece712/tutorial%20on%20PLS%20and%20PCA.pdf Geladi, P., Kowalski, B. R. Partial least-squares regression: a tutorial. ''Anal. Chim. Acta.'' '''1986''', 185: 1-17.] 
[2] [http://www.iasbs.ac.ir/chemistry/chemometrics/history/4th/5.pdf Wold, S., Sjoestroem, M., Eriksson, L. PLS-regression: a basic tool in chemometrics. ''Chemometr. Intell. Lab.'' '''2001''', 58: 109-130.] 
====Process analytical technology (PAT)====
[1] [https://link.springer.com/article/10.1007%2Fs12010-012-9950-y Challa, S., Potumarthi, R. Chemometrics-based process analytical technology (PAT) tools: applications and adaptation in pharmaceutical and biopharmaceutical industries. ''Appl. Biochem. Biotechnol.'' '''2013''', 169: 66–76.] 
====Guidelines for NMR validation====
[1] [http://www.eurolab.org/documents/NMR%20Val%20Guideline%20II%20V6.pdf Schönberger, T., Monakhova, Y. B., Lachenmeier, D.W., Walch, S., Kuballa, T., et al.'' Guide to NMR Method Development and Validation – Part II: Multivariate data analysis.'' Eurolab Technical Report No. 01/2015.] 

===Reviews and Applications of Chemometrics to NMR ===
[1] [https://doi.org/10.1016/j.sajb.2012.04.001 Heyman, H. M., Meyer, J. J. M. NMR-based metabolomics as a quality control tool for herbal products. ''S. Afr. J. Bot.'' '''2012''', 82: 21-32.] 
[2] [https://doi.org/10.1016/j.talanta.2014.02.003 Kumar, N., Bansal, A., Sarma, G. S., Rawal, R. K. Chemometrics tools used in analytical chemistry: an overview. ''Talanta'' '''2014''', 123: 186-199.] 
[3] [http://www.sciencedirect.com/science/article/pii/S007965651100032X?via%3Dihub McKenzie, J. S., Donarski, J. A., Wilson, J. C., Charlton, A. J. Analysis of complex mixtures using high-resolution nuclear magnetic resonance spectroscopy and chemometrics. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 59: 336-359.] 
[4] [http://www.sciencedirect.com/science/article/pii/S0079656514000600 Brennan, L. NMR-based metabolomics: from sample preparation to applications in nutrition research. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2014''', 83: 42-49.] 
[5] [http://www.sciencedirect.com/science/article/pii/S0079656510000932?via%3Dihub Simpson, A. J., McNally, D. G., Simpson, M. J. NMR spectroscopy in environmental research: from molecular interactions to global processes. ''Prog. Nucl. Magn. Reson. Spectrosc.'' '''2011''', 58: 97-175.] 
[6] [http://www.sciencedirect.com/science/article/pii/S0731708510007302 Malet-Martino, M., Holzgrabe, U. NMR techniques in biomedical and pharmaceutical analysis. ''J. Pharm. Biomed. Anal.'' '''2011''', 55: 1-15.] 
[7] [http://pubs.rsc.org/en/content/articlelanding/2002/an/b208254n/unauth#!divAbstract Holmes, E., Antti, H. Chemometric contributions to the evolution of metabonomics: mathematical solutions to characterising and interpreting complex biological NMR spectra. ''Analyst'' '''2002''', 127: 1549-57.] 
[8] [http://www.sciencedirect.com/science/article/pii/S1043452610590041?via%3Dihub Consonni, R., Cagliani, L. R. Nuclear magnetic resonance and chemometrics to assess geographical origin and quality of traditional food products. ''Adv. Food. Nutr. Res.'' '''2010''', 59: 87-165.] 

===Useful Links===
====General software====
[https://www.mathworks.com/products/matlab.html MATLAB] 
[https://www.r-project.org/ R computing system] 
[http://www.camo.com/rt/Products/Unscrambler/unscrambler.html Unscrambler] 
[http://www.chimiometrie.fr/saisir_webpage.html SAISIR] 
[http://umetrics.com/products/simca SIMCA-P] 
[https://www.bruker.com/products/mr/nmr/nmr-software/nmr-software/amix/overview.html AMIX] 
[https://infometrix.com/pirouette/ Pirouette] 
[http://ufla.br/chemoface/ Chemoface] 

====Specialized chemometric methods====
[http://www.cid.csic.es/homes/rtaqam/ MCR] 
[https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA MILCA and SNICA] 
[http://read.pudn.com/downloads6/sourcecode/math/22123/image_mva_0/simplisma.m_.htm SIMPLISMA] 
[http://perso.telecom-paristech.fr/~cardoso/guidesepsou.html JADE] 
[http://www.cs.umass.edu/~elm/ICA/ RADICAL] 
[http://www.cis.hut.fi/projects/ica/fastica/ FastICA] 

====Multi-block Toolbox for Matlab====
http://www.models.life.ku.dk/~courses/MBtoolbox/mbtmain.htm 

====Public data sets for multivariate data analysis====
http://www.models.life.ku.dk/datasets 
https://www.ucl.ac.uk/ion/departments/sobell/Research/RLemon/MILCA/MILCA 

If you have any questions and/or concerns, please contact [[User:Yulia Monakhova|Yulia Monakhova]] [mailto:monakhova@spectralservice.de Email] or [[User:Kim Colson|Kim Colson]] [mailto:kim_colson@bruker.com Email].