Advanced Preprocessing: Sample Normalization - Revision history

imported>Donal: /* Normalize */

2017-03-07T16:41:08Z

Normalize

← Older revision		Revision as of 08:41, 7 March 2017
Line 30:		Line 30:
	\|-		\|-
	\| 2-Norm \|\|		\| 2-Norm \|\|
	Normalize to the sum of the squared value of all variables for the given sample. Returns a vector of unit length (length = 1). A form of weighted normalization where larger values are weighted more heavily in the scaling.		Normalize to the square root of the sum of the squared value of all variables for the given sample. Returns a vector of unit length (length = 1). A form of weighted normalization where larger values are weighted more heavily in the scaling.
	\|\| <math>w_{i}=\left(\sum_{j=1}^{n}x_{i,j}^{2}\right)^{\frac{1}{2}}</math>		\|\| <math>w_{i}=\left(\sum_{j=1}^{n}x_{i,j}^{2}\right)^{\frac{1}{2}}</math>
	\|-		\|-

imported>Bob: /* Normalize */

2017-03-07T15:30:18Z

Normalize

← Older revision		Revision as of 07:30, 7 March 2017
Line 31:		Line 31:
	\| 2-Norm \|\|		\| 2-Norm \|\|
	Normalize to the sum of the squared value of all variables for the given sample. Returns a vector of unit length (length = 1). A form of weighted normalization where larger values are weighted more heavily in the scaling.		Normalize to the sum of the squared value of all variables for the given sample. Returns a vector of unit length (length = 1). A form of weighted normalization where larger values are weighted more heavily in the scaling.
	\|\| <math>w_{i}=\sum_{j=1}^{n}x_{i,j}^{2}</math>		\|\| <math>w_{i}=\left(\sum_{j=1}^{n}x_{i,j}^{2}\right)^{\frac{1}{2}}</math>
	\|-		\|-
	\| Inf-Norm \|\|		\| Inf-Norm \|\|

imported>Jeremy at 18:37, 27 July 2011

2011-07-27T18:37:15Z

← Older revision		Revision as of 10:37, 27 July 2011
Line 49:		Line 49:
	[[Image:App_normalize.png\|\|\|]]		[[Image:App_normalize.png\|\|\|]]

	From the Preprocessing ~~GUI~~, the only setting associated with this method is the type of normalization (1-norm, 2-norm or inf-norm). There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing window, the only setting associated with this method is the type of normalization (1-norm, 2-norm or inf-norm). There is currently no option to perform this normalization based on anything other than all selected variables.

	From the command line, this method is performed using the [[normaliz]] function (note the unusual spelling of the function).		From the command line, this method is performed using the [[normaliz]] function (note the unusual spelling of the function).
Line 77:		Line 77:
	[[Image:App_snv_2.png\|\|\|]]		[[Image:App_snv_2.png\|\|\|]]

	From the Preprocessing ~~GUI~~, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing window, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.
	From the command line, this method is performed using the [[snv]] function.		From the command line, this method is performed using the [[snv]] function.

Line 113:		Line 113:
	The first line creates a matrix of ten spectra. The second multiplies them by a random number between 0 and 1 and adds a random offset. The function mscorr is then used to correct the spectra using the mean of the first twenty spectra as the reference. The outputs of mscorr are the corrected spectra (sx), and the intercepts (alpha), and slope (beta) for the fit of each spectrum to the reference.		The first line creates a matrix of ten spectra. The second multiplies them by a random number between 0 and 1 and adds a random offset. The function mscorr is then used to correct the spectra using the mean of the first twenty spectra as the reference. The outputs of mscorr are the corrected spectra (sx), and the intercepts (alpha), and slope (beta) for the fit of each spectrum to the reference.

	From the Preprocessing ~~GUI~~, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in the last equation above) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing window, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in the last equation above) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.

	From the command line, this method is performed using the [[mscorr]] function. The extended multiplicative scatter correction (EMSC) algorithm is a patented algorithm which can be purchased separately. The algorithm and use are described in the [[emscorr]] documentation pages.		From the command line, this method is performed using the [[mscorr]] function. The extended multiplicative scatter correction (EMSC) algorithm is a patented algorithm which can be purchased separately. The algorithm and use are described in the [[emscorr]] documentation pages.

imported>Jeremy at 18:10, 22 September 2010

2010-09-22T18:10:59Z

← Older revision		Revision as of 10:10, 22 September 2010
Line 44:		Line 44:
	: <math>\mathbf{x}_{i,norm}=\mathbf{x}_{i}w_{i}^{-1}</math>		: <math>\mathbf{x}_{i,norm}=\mathbf{x}_{i}w_{i}^{-1}</math>
	An example using the 1-norm on near infrared spectra is shown in the figure below. These spectra were measured as 20 replicates of 5 synthetic mixtures of gluten and starch (Martens, Nielsen, and Engelsen, 2003) In the original data (top plot), the five concentrations of gluten and starch are not discernable because of multiplicative and baseline effects among the 20 replicate measurements of each mixture. After normalization using a 1-norm (bottom plot), the five mixtures are clearly observed in groups of 20 replicate measurements each.		An example using the 1-norm on near infrared spectra is shown in the figure below. These spectra were measured as 20 replicates of 5 synthetic mixtures of gluten and starch (Martens, Nielsen, and Engelsen, 2003) In the original data (top plot), the five concentrations of gluten and starch are not discernable because of multiplicative and baseline effects among the 20 replicate measurements of each mixture. After normalization using a 1-norm (bottom plot), the five mixtures are clearly observed in groups of 20 replicate measurements each.

			'''Figure''': Effect of normalization on near-IR spectra of five synthetic gluten and starch mixtures. Original spectra (top plot) and spectra after 1-norm normalization (bottom plot) are shown.

	[[Image:App_normalize.png\|\|\|]]		[[Image:App_normalize.png\|\|\|]]

	~~'''Figure''': Effect of normalization on near-IR spectra of five synthetic gluten and starch mixtures. Original spectra (top plot) and spectra after 1-norm normalization (bottom plot) are shown.~~

	From the Preprocessing GUI, the only setting associated with this method is the type of normalization (1-norm, 2-norm or inf-norm). There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing GUI, the only setting associated with this method is the type of normalization (1-norm, 2-norm or inf-norm). There is currently no option to perform this normalization based on anything other than all selected variables.
Line 64:		Line 64:

	This normalization approach is weighted towards considering the values that deviate from the individual sample mean more heavily than values near the mean. Consider the example Raman spectrum in the figure below. The horizontal line at intensity 0.73 indicates the mean of the spectrum. The dashed lines at 1.38 and 0.08 indicate one standard deviation away from the mean. In general, the normalization weighting for this sample is driven by how far all the points deviate from the mean of 0.73. Considering the square term in the equation above and the histogram on the left of the figure, it is clear that the high intensity points will contribute greatly to this normalization.		This normalization approach is weighted towards considering the values that deviate from the individual sample mean more heavily than values near the mean. Consider the example Raman spectrum in the figure below. The horizontal line at intensity 0.73 indicates the mean of the spectrum. The dashed lines at 1.38 and 0.08 indicate one standard deviation away from the mean. In general, the normalization weighting for this sample is driven by how far all the points deviate from the mean of 0.73. Considering the square term in the equation above and the histogram on the left of the figure, it is clear that the high intensity points will contribute greatly to this normalization.

			'''Figure''': Eample Raman spectrum (right plot) and corresponding intensity histogram (left plot). The mean of the spectrum is shown as a dashed line at intensity 0.73; one standard deviation above and below this mean are shown at intensities 1.38 and 0.08 and indicated by the arrow.

	[[Image:App_snv_1.png\|\|\|]]		[[Image:App_snv_1.png\|\|\|]]

	'''Figure''': Eample Raman spectrum (right plot) and corresponding intensity histogram (left plot). The mean of the spectrum is shown as a dashed line at intensity 0.73; one standard deviation above and below this mean are shown at intensities 1.38 and 0.08 and indicated by the arrow.

	This approach is a very empirical normalization method in that one seldom expects that variables for a given sample should deviate about their mean in a normal distribution with unit variance (except in the case where the primary contribution to most of the variables is noise and the variables are all in the same units). When much of the signal in a sample is the same in all samples, this method can do very well. However, in cases where the overall signal changes significantly from sample to sample, problems may occur. In fact, it is quite possible that this normalization can lead to non-linear responses to what were otherwise linear responses. SNV should be carefully compared to other normalization methods for quantitative models.		This approach is a very empirical normalization method in that one seldom expects that variables for a given sample should deviate about their mean in a normal distribution with unit variance (except in the case where the primary contribution to most of the variables is noise and the variables are all in the same units). When much of the signal in a sample is the same in all samples, this method can do very well. However, in cases where the overall signal changes significantly from sample to sample, problems may occur. In fact, it is quite possible that this normalization can lead to non-linear responses to what were otherwise linear responses. SNV should be carefully compared to other normalization methods for quantitative models.

	The figure below shows the result of SNV on the gluten and starch mixtures described earlier. Comparing the SNV results to the original spectra and 1-norm spectra shown in the "normalize" section above, it is obvious that SNV gives tighter groupings of the replicate measurements. In fact, SNV was originally developed for NIR data of this sort, and it behaves well with this kind of response.		The figure below shows the result of SNV on the gluten and starch mixtures described earlier. Comparing the SNV results to the original spectra and 1-norm spectra shown in the "normalize" section above, it is obvious that SNV gives tighter groupings of the replicate measurements. In fact, SNV was originally developed for NIR data of this sort, and it behaves well with this kind of response.

			'''Figure''': Effect of SNV normalization on near-IR spectra measured of five synthetic gluten and starch mixtures.

	[[Image:App_snv_2.png\|\|\|]]		[[Image:App_snv_2.png\|\|\|]]

	~~'''Figure''': Effect of SNV normalization on near-IR spectra measured of five synthetic gluten and starch mixtures.~~

	From the Preprocessing GUI, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing GUI, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.
Line 98:		Line 98:

	The figure below shows an example of MSC using NIR spectra of synthetic gasoline samples with added baseline and scaling effects. The spectra with artificial baseline and scaling effects are shown in the top plot. The corrected spectra are shown in the bottom plot. The corrected spectra were brought to within 0.03 absorbance units of the original spectra (not shown).		The figure below shows an example of MSC using NIR spectra of synthetic gasoline samples with added baseline and scaling effects. The spectra with artificial baseline and scaling effects are shown in the top plot. The corrected spectra are shown in the bottom plot. The corrected spectra were brought to within 0.03 absorbance units of the original spectra (not shown).

			'''Figure''': Example of Multiplicative Scatter Correction of artificial baseline and scaling effects.

	[[Image:App_snv_3.png\|\|\|]]		[[Image:App_snv_3.png\|\|\|]]

	~~'''Figure''': Example of Multiplicative Scatter Correction of artificial baseline and scaling effects.~~

	The simulated baseline addition and subsequent correction can be done using the following commands:		The simulated baseline addition and subsequent correction can be done using the following commands:

imported>Jeremy: /* Normalize */

2010-07-07T23:08:04Z

Normalize

← Older revision		Revision as of 15:08, 7 July 2010
Line 38:		Line 38:
	\|}		\|}

	* Where, in each case, wi is the normalization weight for sample i, xi is the vector of observed values for the given sample, j is the variable number, and n is the total number of variables (columns of x).		<tt></tt>* Where, in each case, wi is the normalization weight for sample i, xi is the vector of observed values for the given sample, j is the variable number, and n is the total number of variables (columns of x).

	The weight calculated for a given sample is then used to calculate the normalized sample, <math>\mathbf{x}_{i,norm}</math>, using:		The weight calculated for a given sample is then used to calculate the normalized sample, <math>\mathbf{x}_{i,norm}</math>, using:
Line 45:		Line 45:
	An example using the 1-norm on near infrared spectra is shown in the figure below. These spectra were measured as 20 replicates of 5 synthetic mixtures of gluten and starch (Martens, Nielsen, and Engelsen, 2003) In the original data (top plot), the five concentrations of gluten and starch are not discernable because of multiplicative and baseline effects among the 20 replicate measurements of each mixture. After normalization using a 1-norm (bottom plot), the five mixtures are clearly observed in groups of 20 replicate measurements each.		An example using the 1-norm on near infrared spectra is shown in the figure below. These spectra were measured as 20 replicates of 5 synthetic mixtures of gluten and starch (Martens, Nielsen, and Engelsen, 2003) In the original data (top plot), the five concentrations of gluten and starch are not discernable because of multiplicative and baseline effects among the 20 replicate measurements of each mixture. After normalization using a 1-norm (bottom plot), the five mixtures are clearly observed in groups of 20 replicate measurements each.

	[[Image:App_normalize.png\|\|\|]] ~~Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4~~		[[Image:App_normalize.png\|\|\|]]

	'''Figure''': Effect of normalization on near-IR spectra of five synthetic gluten and starch mixtures. Original spectra (top plot) and spectra after 1-norm normalization (bottom plot) are shown.		'''Figure''': Effect of normalization on near-IR spectra of five synthetic gluten and starch mixtures. Original spectra (top plot) and spectra after 1-norm normalization (bottom plot) are shown.

imported>Jeremy: moved Sample normalization to Advanced Preprocessing: Sample Normalization

2010-07-07T23:05:56Z

moved Sample normalization to Advanced Preprocessing: Sample Normalization

← Older revision	Revision as of 15:05, 7 July 2010
(No difference)

imported>Jeremy: /* MSC (Multiplicative Scatter Correction) */

2010-07-07T23:04:53Z

MSC (Multiplicative Scatter Correction)

← Older revision		Revision as of 15:04, 7 July 2010
Line 115:		Line 115:
	From the Preprocessing GUI, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in the last equation above) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing GUI, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in the last equation above) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.

	From the command line, this method is performed using the [[mscorr]] function.		From the command line, this method is performed using the [[mscorr]] function. The extended multiplicative scatter correction (EMSC) algorithm is a patented algorithm which can be purchased separately. The algorithm and use are described in the [[emscorr]] documentation pages.

imported>Jeremy: /* MSC (Multiplicative Scatter Correction) */

2010-07-07T23:03:45Z

MSC (Multiplicative Scatter Correction)

← Older revision		Revision as of 15:03, 7 July 2010
Line 113:		Line 113:
	The first line creates a matrix of ten spectra. The second multiplies them by a random number between 0 and 1 and adds a random offset. The function mscorr is then used to correct the spectra using the mean of the first twenty spectra as the reference. The outputs of mscorr are the corrected spectra (sx), and the intercepts (alpha), and slope (beta) for the fit of each spectrum to the reference.		The first line creates a matrix of ten spectra. The second multiplies them by a random number between 0 and 1 and adds a random offset. The function mscorr is then used to correct the spectra using the mean of the first twenty spectra as the reference. The outputs of mscorr are the corrected spectra (sx), and the intercepts (alpha), and slope (beta) for the fit of each spectrum to the reference.

	From the Preprocessing GUI, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in equation ~~7-8~~) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing GUI, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in the last equation above) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.

	From the command line, this method is performed using the [[mscorr]] function.		From the command line, this method is performed using the [[mscorr]] function.

imported>Jeremy: /* MSC (Multiplicative Scatter Correction) */

2010-07-07T23:02:57Z

MSC (Multiplicative Scatter Correction)

← Older revision		Revision as of 15:02, 7 July 2010
Line 81:		Line 81:

	===MSC (Multiplicative Scatter Correction)===		===MSC (Multiplicative Scatter Correction)===

			Multiplicative signal correction (MSC) is a relatively simple processing step that attempts to account for scaling effects and offset (baseline) effects (Martens and Næs, 1989). This correction is achieved by regressing a measured spectrum against a reference spectrum and then correcting the measured spectrum using the slope (and possibly intercept) of this fit. Specifically, MSC follows this procedure: Define x as a column vector corresponding to a spectrum to be standardized and r as a vector corresponding to the reference spectrum (often this is the mean spectrum of the calibration data set). The vectors are most often mean-centered according to:

			:<math>\mathbf{x}_c=\mathbf{x}-\bar{x}\mathbf{1}</math>

			:<math>\mathbf{r}_c=\mathbf{r}-\bar{r}\mathbf{1}</math>

			where <math>x_c</math> and <math>r_c</math> are the mean centered vectors, <math>\bar{x}</math> and <math>\bar{r}</math> are the respective means, and <math>\mathbf{1}</math> is a vector of ones. The unknown multiplicative factor b is determined using:

			:<math>\mathbf{r}_cb=\mathbf{x}_c</math>
			:<math>b=(\mathbf{r}_c^T\mathbf{r}_c)^{-1}\mathbf{r}_c^T\mathbf{x}_c</math>

			and the corrected spectrum <math>\hat{\mathbf{x}}</math> is then given by:

			:<math>\hat{\mathbf{x}}=\mathbf{x}_c/b+\bar{r}\mathbf{1}</math>

			The figure below shows an example of MSC using NIR spectra of synthetic gasoline samples with added baseline and scaling effects. The spectra with artificial baseline and scaling effects are shown in the top plot. The corrected spectra are shown in the bottom plot. The corrected spectra were brought to within 0.03 absorbance units of the original spectra (not shown).

			[[Image:App_snv_3.png\|\|\|]]

			'''Figure''': Example of Multiplicative Scatter Correction of artificial baseline and scaling effects.

			The simulated baseline addition and subsequent correction can be done using the following commands:

			<pre>
			» bspec = spec1.data(21:30,:);
			» for i = 1:10, bspec(i,:) = bspec(i,:)*rand(1) + randn(1); end
			» [sx,alpha,beta] = mscorr(bspec,mean(spec1.data(1:20,:)));
			</pre>

			The first line creates a matrix of ten spectra. The second multiplies them by a random number between 0 and 1 and adds a random offset. The function mscorr is then used to correct the spectra using the mean of the first twenty spectra as the reference. The outputs of mscorr are the corrected spectra (sx), and the intercepts (alpha), and slope (beta) for the fit of each spectrum to the reference.

			From the Preprocessing GUI, there are two settings which can be modified: intercept and spectral mode. The intercept option controls whether or not the intercept (i.e., <math>\bar{r}</math> in equation 7-8) is used. The spectral mode setting defaults to 2 and can only be modified when x is a multiway matrix. This setting defines the data mode which is associated with the spectral dimension (i.e., the mode which should be used in the regression). There is currently no option to perform this normalization based on anything other than all selected variables.

			From the command line, this method is performed using the [[mscorr]] function.

imported>Jeremy: /* SNV (Standard Normal Variate) */

2010-07-07T22:51:58Z

SNV (Standard Normal Variate)

← Older revision		Revision as of 14:51, 7 July 2010
Line 68:		Line 68:

	'''Figure''': Eample Raman spectrum (right plot) and corresponding intensity histogram (left plot). The mean of the spectrum is shown as a dashed line at intensity 0.73; one standard deviation above and below this mean are shown at intensities 1.38 and 0.08 and indicated by the arrow.		'''Figure''': Eample Raman spectrum (right plot) and corresponding intensity histogram (left plot). The mean of the spectrum is shown as a dashed line at intensity 0.73; one standard deviation above and below this mean are shown at intensities 1.38 and 0.08 and indicated by the arrow.

	~~Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4~~

	This approach is a very empirical normalization method in that one seldom expects that variables for a given sample should deviate about their mean in a normal distribution with unit variance (except in the case where the primary contribution to most of the variables is noise and the variables are all in the same units). When much of the signal in a sample is the same in all samples, this method can do very well. However, in cases where the overall signal changes significantly from sample to sample, problems may occur. In fact, it is quite possible that this normalization can lead to non-linear responses to what were otherwise linear responses. SNV should be carefully compared to other normalization methods for quantitative models.		This approach is a very empirical normalization method in that one seldom expects that variables for a given sample should deviate about their mean in a normal distribution with unit variance (except in the case where the primary contribution to most of the variables is noise and the variables are all in the same units). When much of the signal in a sample is the same in all samples, this method can do very well. However, in cases where the overall signal changes significantly from sample to sample, problems may occur. In fact, it is quite possible that this normalization can lead to non-linear responses to what were otherwise linear responses. SNV should be carefully compared to other normalization methods for quantitative models.

	The figure below shows the result of SNV on the gluten and starch mixtures described earlier. Comparing the SNV results to the original spectra and 1-norm spectra shown in the "normalize" section above, it is obvious that SNV gives tighter groupings of the replicate measurements. In fact, SNV was originally developed for NIR data of this sort, and it behaves well with this kind of response.		The figure below shows the result of SNV on the gluten and starch mixtures described earlier. Comparing the SNV results to the original spectra and 1-norm spectra shown in the "normalize" section above, it is obvious that SNV gives tighter groupings of the replicate measurements. In fact, SNV was originally developed for NIR data of this sort, and it behaves well with this kind of response.
	~~Normal 0 false false false EN-US X-NONE X-NONE MicrosoftInternetExplorer4~~ '''Figure''': Effect of SNV normalization on near-IR spectra measured of five synthetic gluten and starch mixtures.
			[[Image:App_snv_2.png\|\|\|]]

			'''Figure''': Effect of SNV normalization on near-IR spectra measured of five synthetic gluten and starch mixtures.

	From the Preprocessing GUI, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.		From the Preprocessing GUI, the only setting associated with this method is the offset. There is currently no option to perform this normalization based on anything other than all selected variables.