Validity of Career Predictions

Introduction

CareerExplorer predicts fit with careers based on a person's interests, personality, needs, and preferences. Predictive validation asks the question “How well does it work?” In other words, to what degree can CareerExplorer predict a person's reactions to careers?

Evidence for validity

We test the validity of our predictions with every user. After completing CareerExplorer's assessment, users are invited to make “thumbs up” or “thumbs down” decisions on careers based on one-page career descriptions. For users, this process helps create a shortlist of careers worth deeper exploration. For CareerExplorer, we use the data to test our predictions.

The key is that we include a range of careers, including some we think may be modest or poor fits, presented in a random order. If our assessment is valid, our estimate of career fit should predict which careers receive a “thumbs up”.

Findings

Results strongly support the validity of CareerExplorer's predictions. Roughly 90% of the variance in “thumbs up” decisions can be accounted for by CareerExplorer's career fit predictions.

When presented with their best career matches, users have a 63% chance of giving the career a “thumbs up”. By the 90th fit percentile (i.e., the career fits better than 90% of the 450 careers), rate of receiving a “thumbs up” drops to 24%. At the 70th percentile, the rate falls to 13%. At the median point (half of careers fit better, half worse), only 5% of careers presented to users receive “thumbs up”. Below the 20th percentile, careers almost never receive a “thumbs up”. The following figure displays the “thumbs up” rate at each fit level.

Chart showing very strong correlation between CareerExplorer match score, and how often a career is liked by the user
CareerExplorer's prediction of career fit
(percentile out of 450 careers)		 Percent of careers
receiving a thumbs up		 Number of thumbs up
vs. down decisions		 Number of respondents
100 62.8% 9257 2794
99 52.4% 9175 2797
98 45.9% 9118 2798
97 40.5% 8936 2770
96 34.9% 6481 2455
95 31.4% 5949 2206
94 29.5% 5086 2156
93 28.1% 4458 2016
92 26.3% 4178 1963
91 25.9% 3769 1886
90 23.9% 3714 1912
89 23.3% 3836 2123
88 23.4% 3987 2205
87 22.9% 4030 2222
86 22.2% 3982 2241
85 22.1% 3946 2269
84 20.7% 3848 2196
83 19.3% 3903 2213
82 19.8% 3833 2221
81 18.5% 3866 2224
80 18.4% 3893 2234
79 17.6% 3789 2216
78 17.0% 3845 2227
77 15.7% 3283 1986
76 14.8% 3173 1930
75 15.3% 3214 1950
74 14.6% 3271 1990
73 13.8% 3235 1967
72 13.9% 3190 1966
71 13.2% 3216 1963
70 13.2% 3267 1942
69 12.3% 3189 1938
68 12.0% 3233 1978
67 11.9% 3139 1917
66 10.8% 2478 1690
65 6.3% 941 545
64 8.1% 881 501
63 6.0% 999 538
62 6.6% 966 531
61 7.8% 990 556
60 7.2% 971 556
59 6.7% 868 512
58 7.8% 901 573
57 6.1% 834 515
56 7.8% 845 537
55 6.1% 836 545
54 7.0% 817 548
53 6.7% 774 525
52 6.2% 790 553
51 7.4% 739 516
50 5.1% 746 522
49 6.1% 737 525
48 3.6% 759 535
47 6.1% 748 541
46 5.4% 736 522
45 5.1% 693 520
44 5.2% 687 525
43 3.5% 684 542
42 4.6% 653 506
41 5.5% 642 506
40 3.4% 676 523
39 4.7% 706 563
38 3.4% 680 526
37 1.9% 257 95
36 3.5% 226 69
35 1.8% 219 57
34 1.4% 212 55
33 1.5% 199 54
32 5.4% 186 47
31 3.8% 182 47
30 1.1% 181 44
29 2.3% 176 45
28 1.7% 174 45
27 1.1% 176 43
26 0.6% 164 43
25 1.2% 161 41
24 2.0% 151 36
23 0.7% 148 37
22 1.3% 153 37
21 1.4% 148 36
20 1.4% 141 34
19 0.0% 142 34
18 1.4% 141 35
17 0.0% 144 35
16 0.0% 137 33
15 0.0% 136 34
14 0.7% 139 34
13 0.8% 127 30
12 0.0% 126 29
11 0.8% 120 27
10 0.0% 124 29
9 0.8% 120 28
8 2.5% 122 29
7 0.0% 124 28
6 0.0% 117 27
5 0.0% 123 28
4 0.8% 120 29
3 0.0% 114 27
2 0.9% 110 26
1 0.0% 111 26

More evidence to come

We're currently conducting other studies and planning even more. The study described on this page relies on respondent judgment for both the predictors and criteria. Future studies will use other types of criteria to test predictions, and explore how well CareerExplorer's assessment corresponds to other measures of similar constructs.