Predicting Criteria
Personality traits and other characteristics can be used to predict real life outcomes. For example, extraversion can be used to predict success as a salesperson, or IQ can be used to predict future grade point average.
Correlation Coefficients as Descriptions of Predictions
These predictions are often described in terms of a correlation coefficient -- a statistic that indicates the strength and direction of a relationship between two variables. Correlation coefficients range from r = -1.0 to 1.0. An r of 1 indicates a perfect relationship. An r of 0 indicates a complete lack of (a linear) relationship, and an r = -1.0 indicates a perfect inverse relationship.
The Language "Predicting the Variance..."
Many personality predictions occur between r's from -.40 to +.40. That is, they do not range up into the high or perfect range of prediction.
Psychologists have spent considerable effort trying to characterize the strength of correlations in the r = -.40 to .40 range. Generally they speak in terms of "explaining a portion of the variance" of a criterion. For example, extraversion predicts 4% of the variance of success at sales, or intelligence predicts 25% of the variance of grades in school.
Most people (including most psychologists) have difficulty understanding exactly what "explaining a percentage of the variance" refers to. In fact, the explanation tends to be rather technical. The "variance" here, refers to the average deviation of each person from the mean -- but where the average deviation is measured as the average of the square root of the squared deviations, and the deviations represent the difference of each person from the sample mean. Follow? Perhaps, like most people, you don't find this jumps out at you as a great metric. Hence, the Binomial Effect Size Display.
The Binomial Effect Size Display (BESD)
The best way I know of to explain "explaining the variance," is the binomial effect size display table (Rosenthal & Rubin, 1982).
Here is how it works (without going into the math too much).
First, below, find a situation in which a test makes no prediction. Whether or not you score high or low on the test is unrelated to whether you will be highly successful or just okay at a particular job. The table immediately below shows the situation for a group of 200 people.
Table 1: A Prediction to the Crtierion of r = .00 (No Prediction) |
|
Performance Okay |
Highly Successful |
Total |
Low Test Score |
50 |
50 |
100 |
High Test Score |
50 |
50 |
100 |
Total |
100 |
100 |
200 |
Now, say the test predicts at the r = .40 level -- "explaining 16% of the variance." The numbers for this group of 200 people change like this…
Table 2: A Prediction to the Crtierion of r = .40 |
|
Performance Okay |
Highly Successful |
Total |
Low Test Score |
70 |
30 |
100 |
High Test Score |
30 |
70 |
100 |
Total |
100 |
100 |
200 |
It is a rather dramatic change. Going up to an r = .5, or "explaining 25% of the variance" -- something that intelligence tests sometimes do -- yields a chart like this:
Table 3: A Prediction to the Crtierion of r = .50 |
|
Performance Okay |
Highly Successful |
Total |
Low Test Score |
75 |
25 |
100 |
High Test Score |
25 |
75 |
100 |
Total |
100 |
100 |
200 |
Now, some people look at a result such as that in Table 3 and they notice that people still will be mis-classified. They are right.
On that basis, however, they argue that tests ought not to be used, because they are imperfect. In this case they are (to me) quite wrong. Throwing out tests throws out crucial information that can lead to far more successess and far fewer failures than otherwise.
To make this clearer consider a case in which a real organization can select 100 people (of 200) and place them into positions. Still looking at Table 3, note that 150 have been correctly classified. It is, however, true also that, in the case of an r = .50, 50 people are misclassified -- 25 of whom are unfairly allowed to obtain a position, and 25 of whom are unfairly excluded. This is unfortunate, imperfect, and problematic, but it is (as is shown below) an improvement over the case of not using a test. That is, life circumstances are often difficult, but the test creates a relative, though imperfect, improvement.
Compare the case illustrated in Table 3 to a case where no test is used, as shown in Table 4. There, the 100 people are essentially selected by chance to fill the positions, and 100 are excluded essentially by chance. Under these conditions, 50 people will be unfairly excluded, and 50 unfairly selected. That is, the number of people unfairly treated doubles. This is why tests are useful for prediction and selection.
Random assignment, in other words, creates more selection failure.
Table 4: No Prediction |
|
Performance Okay |
Highly Successful |
Total |
| No Test Score, Excluded |
50 |
50 |
100 |
No Test Score, Selected |
50 |
50 |
100 |
Total |
100 |
100 |
200 |
References
Rosenthal, R., & Rubin, D. B. (1982). A simple, general purpose display of magnitude of experimental effect. Journal of Educational Psychology, 74, 166-169.