Can machine learning algorithms perform better than multiple linear regression in predicting nitrogen excretion from lactating dairy cows | Scientific…

All the experiments were conducted at the Agri-Food and Biosciences Institute (AFBI) farm at Hillsborough, County Down, UK. All the experiments and procedures complied with the requirements of the UK Animals (Scientific Procedures) Act 1986 and were approved by the AFBI Hillsborough Ethical Review Group. All the experiments were performed in accordance with relevant guidelines and regulations (following the ARRIVE guidelines26).

Data used were collated from 43 total diet digestibility studies with 951 lactating dairy cows undertaken at Agri-Food and Biosciences Institute in Northern Ireland over a period of 26years (19902015). The data from studies undertaken between 1990 and 2002 were used as the training dataset (n=564) and undertaken between 2005 and 2015 as the testing dataset (n=387). The training data were used to develop prediction models for MN using MLR and the three selected machine learning algorithms (ANN, RFR and SVR). These new models were then tested for their predictive performance using the training dataset by tenfold cross validation. The testing dataset were used for the independent evaluation and comparison of predictive ability of different modeling approaches. The information of the two datasets on numbers of experiments, cow genotypes and forage types offered are presented in Table 10. Data on live weight, milk production, feed intake, N intake and outputs are presented in Table 11. The datasets used in the present study showed a various cow genetic merit and a broad range in LW (379781kg), MY (5.140.2kg/d), total dry matter intake (7.5426.6kg/d), FP (0.211.00%), DNC (19.038.0g/kg DM), diet metabolizable energy concentration (DMEC, 9.6819.4MJ/kg DM) and NI (155874g/d), which represents typical dairy production conditions managed within grassland-based dairy systems in the West and North Europe.

Cows were housed in free-stall cubicle accommodation for at least 20 d before commencing digestibility trials in metabolism units for 8 d with feed intake, milk production and feces and urine collected during the final 6 d. Throughout the whole experiment, cows were offered experimental diets ad libitum and had free access to water. During the final 6 d, the following measurements for each individual cows were carried out to generate total digestibility data used in the present study. Forages and concentrates offered and refused were recorded daily and sampled for analysis of feed dry matter (DM), N concentration and forage proportion. Feces and urine outputs were collected daily and sampled for DM (feces only) and N concentration. Milk yield was recorded daily and sampled for analysis fat, protein and lactose concentrations. Live weight was measured on the first and last days in the metabolism unit. Details in feed intake, feces and urine collection and methods used for analysis of feed, feces, urine and milk samples were described by Yan et al.6.

Because features (variables) in raw data may have different dynamic ranges, which may result in poor model performance, it is recommended to normalize them to make ANN training more efficient by performing normalization process for the raw inputs10. In the present study, all the input data for ANN models were normalized into the interval [0, 1] by performing MinMax normalization technique27 using Eq.(1):

$$X_{norm} = frac{{X {-} X_{min} }}{{X_{max} - X_{min} }}$$

(1)

where Xnorm or X is the normalized or original value, Xmin or Xmax is the minimum or maximum values of the input data.

After finding the optimal tuning parameter, all normalized data for MN obtained by ANN models were denormalized into their original scale using Eq.(2) 27:

$$Y = Y_{norm} * , left( {Y_{max} - , Y_{min} } right) , + , Y_{min}$$

(2)

where Ynorm or Y is the normalized or demoralized value, Ymin or Ymax is the minimum or maximum values of the output data.

Feature selection is an essential step during development of models, which can hugely impact the generalization and predictive ability of models10,28. In the present study, a hybrid knowledge-based and data driven approach was developed and implemented to selecting features. Knowledge in animal science and the process of digestibility trial were applied to diagnosing and removing irrelevant features before the implementing of data driven feature selection process. For instance, the features of feces N output (FN) and urine N output (UN) were excluded from the set of features in present study according to prior background and expert knowledge. Because the data of UN and FN were obtained from analyzing urine and feces samples and then they were summed up and treated as new feature MN, both FN and UN are heavily correlated with MN. Their inclusion in the features list might cause poor generalization performance of the models. Furthermore, the optimal features selected from data driven approach may need to be diagnosed based on background knowledge in animal science according to the scenarios of model application. For instance, several variables (e.g. NI and FP) included in datasets used in this study may not be available in commercial farms. Therefore, alternative feature (concentrate dry matte intake, CDMI) was selected and included into the feature list in this study based on the domain knowledge and then new ANN model suits for commercial farms was developed.

The filter method was applied for feature selection using the Pearson correlation matrix and variance inflation factor (VIF) technique. The first step was to use the Pearson correlation matrix to identify features which might correlate each other for prediction of MN excretion, because using correlated features in models could influence performance of these models with a biased outcome. If two features were heavily correlated, the less important one was removed from the set of features to minimize adverse effects on model performance. Afterwards, the VIF analysis was applied to detect multicollinearity, which has been widely used as a measure of the degree of multicollinearity among input features. A VIF score was calculated for each feature and those with high values were removed. The threshold score for the VIF analysis was 5 and features with a VIF score below this threshold were selected. The VIF score was computed by VIF function in R29.

In the present study, four models based on the MLR ANN, RFR and SVR were developed using the training dataset and these new models were tested using the testing dataset for comparison of their prediction performance for MN outputs in lactating dairy cows (presented later). The MLR with the stepwise procedure for selection of independent variables was used as benchmark model since it is a well-known technique and has been applied for modelling in a wide range of applications in animal science research. Alternative modeling approaches proposed in the present study were ANN, RFR and SVR. To compare the performance, models developed with different approaches and ensure that the same resampling sets were used between calls, the same random number seeds were set prior to perform the process of training, fitting and testing models. All statistical analyses were performed with R29.

The MLR model (Eq.3) selected in the present study for the prediction of MN output was published in 20066 which was developed using the same training dataset listed in Table 2. To improve the estimation of the regression parameters, experiment was included as a random factor during the development of MLR model. The dataset had a large range within each dependent or independent variable, e.g., MN, NI, LW, MY, FP and DNC, which is vital to ensure the development of robust regression model applicable under various farming conditions10.

$${text{MN }}left( {{text{g}}/{text{d}}} right) , = , 0.{text{749 NI }} + , 0.0{text{65 LW }}{-}{ 1}.{text{515 MY }}{-}{ 17}.0$$

(3)

where NI, LW and MY are N intake (g/d), live weight (kg) and milk yield (kg/d), respectively.

In the present study, ANN was fitted using R package neuralnet which was built to train neural networks in the context of regression analyses. The details of ANN training and application of neuralnet were described by Gnther and Fritsch30. Multilayer perceptron networks trained with backpropagation learning algorithms were used and consist of an input layer, hidden layer(s) and an output layer. The input variables were obtained by using the feature selection algorithm described in the section Knowledge-based and data driven feature selection, and the neuron in output layer represents MN. The ANN models were trained based on the selection of training algorithms and learning parameters including the number of hidden layer(s), number of neurons in hidden layer(s), error function, threshold for partial derivatives of the error function as stopping criteria, and activation function etc.. The optimized number of hidden layer(s), number of neuron(s) in the hidden layer(s), learning algorithms, learning rate and other learning parameters were obtained on the basis of prediction performance measured as relative root mean square error (RRMSE, Eq.6) with tenfold cross validation and then the best topology/architecture was finalized.

The RFR is an ensemble machine learning method and a nonparametric technique derived from classification and regression trees which are constructed using a bootstrap aggregating (bagging) method from the training data31. In RFR, prediction is conducted by averaging the individual tree predictions. A detailed description of RFR theory can be found in the report by Breiman32. The RFR was implemented by the randomForest function in the R package (version 3.6.1). To select the optimal hyperparameters for learning algorithm, tuning process was performed based on the R package ranger. The hyperparameters include number of trees to grow (ntree), number of randomly drawn candidate variables (mtry), sample size and node size. Grid search strategy was used to choose the candidate hyperparameter values and the performances of the trained algorithm with different values of the hyperparameters were evaluated as RRMSE (Eq.6) by using tenfold cross validation.

The SVR uses similar principles as support vector machine, a supervised non-parametrical statistical learning technique that uses the kernel functions and the maximum margin algorithm to solve the nonlinear problem33. The detailed theoretical background and description of SVR can be found in the report by Cristianini and Shawe-Taylor34. The SVR model performs the regression estimation by risk minimization where the risk is measured by a loss function. In this study, R package e1071 was used and the svm function was implemented to fit SVR model. The radial basis kernels function, the most commonly used kernels types, was employed in training and predicting process. Parameter tuning was performed by using grid search over supplied parameter ranges and the best combination of parameters (lowest RMSE) were selected. The performance of SVR model was measured as RRMSE (Eq.6) with tenfold cross validation.

The MLR model and the three new models (ANN, RFR and SVR) was developed and compared in terms of their prediction performance for MN outputs in lactating dairy cows based on the datasets listed in Table 2. The predictive performance of models were evaluated using coefficient of determination (R2), root mean square error (RMSE), relative root mean square error (RRMSE) and concordance correlation coefficient (CCC), based on the actual and predicted values. The R2 was calculated using Eq.(4). The RMSE and RRMSE were produced in a tenfold cross validation process (10 RMSE data generated) using Eq.(5)35 and Eq.(6)36, respectively. The concordance correlation coefficient (CCC), a further measure of the agreement between observed and predicted values, was given by Eq.(7)37. The tenfold cross validation was used to evaluate prediction performance of these models (MLR, ANN, RFR and SVR)The obtained RMSE, RRMSE and CCC values (n=10) through the tenfold cross validation were compared among the 4 models using one-way analysis of variance and then followed by Tukeys honest significant difference (HSD) test (=0.05). The same cross validation folds were used for all modeling scenarios to compare cross all of the models performance.

$$R^{2} = 1 - frac{{sum left( {y_{i} - hat{y}} right)^{2} }}{{sum left( {y_{i} - overline{y}} right)^{2} }}$$

(4)

$$RMSE = sqrt { frac{1}{n}mathop sum limits_{i = 1}^{n} left( {y_{i} - hat{y}} right)^{2} }$$

(5)

$$RRMSE = (RMSE/overline{y}) times , 100$$

(6)

$$CCC = frac{{2 cdot ,r cdot ,S_{{widehat{y}}} cdot,S_{y} }}{{S_{{widehat{y}}}^{2} + S_{y}^{2} + left( {mathop sum nolimits_{i = 1}^{n} frac{{left( {y_{i} - widehat{y}} right)}}{n}} right)^{2} }}$$

(7)

where (y_{i}) is actual MN, (widehat{{y_{i} }}) is predicted MN, (overline{y}) is the mean of actual MN and n is the number of observations, r is the Pearson correlation coefficient between (widehat{{y_{i} }}) and (overline{y}), (S_{{hat{y}}}) and (S_{y}) are the respective standard divisions.

See the article here:
Can machine learning algorithms perform better than multiple linear regression in predicting nitrogen excretion from lactating dairy cows | Scientific...