Intelligent systems in accounting, finance and managementassessing predictive performance of ann-based classifiers, страница 18

6.4.  Experiment 4

Here we use non-parametric tests to validate the hypotheses H4 (that the crossover operator has an influence on both the GA-based ANN training and the testing performances; Table X) and H5 (that the generation of effective training and validation sets has an influence on both RT-based ANN training and testing performances). As for the first experiment, we used the real dataset (the original telecom data) without preprocessing the data (first preprocessing approach).

We found a very weak support: two pair-vectors differ significantly at a level of significance of 0.1, namely GAO versus GAM and GAM versus GAU, both in the case of training phase. Also, we found no evidence to differentiate between the three RT mechanisms.

6.5.  Experiment 5

In the first four experiments, when we validate our hypotheses, we relied exclusively on nonparametric tests. We argued that the parametric tests (like t-test, univariate ANOVA, etc.) require the vectors analysed to satisfy different assumptions. For instance, when applying ANOVA one should check the following assumptions: that the observations are independent, that the sample data have a normal distribution, and that scores in different groups have homogeneous variances. The first assumption is satisfied, since all other factors besides preprocessing, distribution and training mechanism that could influence the classifiers’ performances are fixed. For the second assumption we argue that ANOVA is robust against normality assumptions if the sample size is large. Regarding the third assumption, SPSS (the software that we used) incorporates the case when the variances between groups are assumed to be non-equal.

In order to give more strength to our results from the previous experiments and, at the same time, to validate our main hypothesis, we finally performed a three-way ANOVA analysis having as grouping variables the technique used (GAO, GAM, GAA, GAU, RT1, RT2, and RT3), the preprocessing method (PR1, i.e. no preprocessing; PR2, i.e. normalization; PR3, i.e. dividing the variables by the maximum absolute values), and the data distribution (REAL, UNIF, NORM, LOG, and LAP). With the third preprocessing method we obtained values between −1 and +1. We used the vectors’ means to fill in our accuracy rates data. Tables XI and XII include the data we used to perform the three-way ANOVA.

Next, the results of three-way ANOVA for both training and test accuracy rates are shown in Tables XIII and XIV.

As the tables show, all the factors are statistically significant. In other words, they have an individual and combined influence on both training and testing performances. The last column (partial η²) reports the ‘practical’ significance of each term, based upon the ratio of the variation (sum of squares) accounted for by the term to the sum of the variation accounted for by the term and the variation left to error. Larger values of partial η² indicate a greater amount of variation accounted for by the model term, to a maximum of unity. Here, the individual factors and their combinations, while statistically significant, have great effect on classifier accuracy. Consequently, the main hypothesis, i.e. H6, is validated.

In the next three tables we present the pairs’ comparisons for the training performances. The second hypothesis (H2) is validated (Table XV) and ‘normalization’ is the best preprocessing approach, followed in order by ‘maximum absolute values’ and ‘no preprocessing’. Concerning the third hypothesis (H3), the best performance was obtained when data were normally distributed (Table XVI). The next best distribution was that of the real data, followed by uniform, logistic and

Table XI. Accuracy rates for training