Alumni earning, college quality, and the case of the missing correlations

The Chronicle of Higher Education ran a commentary on 5/8/11 that really got me agitated: http://chronicle.com.proxy.cc.uic.edu/article/Want-to-Know-What-Grads-Make-/127421/ 

The commentary, by Richard Vedder, argues that the IRS should work with colleges and universities to compile reports on alumni earnings:

"Students could readily compare colleges not only in terms of costs (based on tuition charges, less expected financial assistance), but also potential future revenues. Companies, including those that rank colleges, could calculate estimated average rates of return on families' investments in college, taking into account such factors as average time to degree and dropout rates"  -- (Vedder, 2011)

I can think of several reasons why this is a bad idea.  Most prominent is that differences in alumni earnings are not driven solely by differences in school quality—there will also be a strong influence from the personal characteristics of the students a school can attract (SAT/ACT score, high-school GPA, SES, etc).  It would be ludicrous, then, to encourage students in the general public to make predictions of 'potential future revenues' that don't take into account their personal attributes.  More plainly, unless you are already 'Harvard material', attending Harvard will not magically confer upon you the earning potential of its other distinguished alums.  Similarly, a student already accepted into Harvard probably won't doom themselves to future poverty by slumming it at Dartmouth.

These considerations suggest an easy improvement to Vedder's proposal: simply take student characteristics into account.  Specifically, you could regress school-average student characteristics onto alumni earnings; the residuals would then reflect true differences in earning potential that are independent of initial differences in student characteristics. 

Although this sounds reasonable, it may not be so simple due to a statistical mystery still unfolding in higher-education studies: a curious lack of correlation between college characteristics and learning gains once student characteristics are accounted for.  As summarized in Pascarella & Terenzi’s masterful second volume of How College Effects Students (2005), there is a large body of empirical evidence examining how student’s basic academic skills improve over the course of college.  The main findings are: a) that gains are fairly modest, and b) that once student characteristics are accounted for, there are no strong and reliable effects of college characteristics.  Specifically, it doesn’t seem to matter if the schools are selective or not, 2-year of 4-year, female-only or co-ed, historically Black or integrated, etc.  Although a few college characteristics have been modestly associated with adjusted learning gains, Pascarella & Terenzi conclude that "between-college effects on the acquisition of subject matter knowledge and academic skills are generally inconsistent and quite small in magnitude" (p. 146).  Take a moment and let that sink in: students do learn in college, and the amount they learn is a function of the skills they bring with them to college, but the specific college they attend does not seem to matter, at least not in a way that we can reliably capture with empirical studies.

This curious case of the missing correlation suggests caution with using alumni earnings to guide college decision-making.  It would be wholly inappropriate to use raw alumni earnings without adjusting for student characteristics, but once these student qualities are factored out the value of alumni earnings may simply vanish.  As is the case with learning gains, the adjustment for student characteristics may leave no consistent or strong relationship between what alumni earn and what school they attended.  Wouldn’t that be a pickle?  Indeed this is exactly the claim made by a UofPhoenix funded and quite controversial report which claims to show that graduates of for-profit institutions end up earning comparable wages to graduates of selective non-profit schools within 10 years of graduation (CoHE summary here: http://chronicle.com/article/article-content/127480/; I haven’t actually read the report or the full article yet).

While I feel this is a compelling critique of the use of alumni income to guide college choices, it has forced me to return to How College Effects Students and contemplate again this statistical equivalent of the dog that didn’t bark in the night: how can it be that college characteristics don’t predict learning gains beyond student characteristics?  Does this mean that all schools are equally good (or bad) at teaching students?  Or, as some would argue, that no learning is occurring at all?  Or is it a methodological issue that is simply obscuring the real impact of college environment?  Perhaps this mystery has already been solved and I’m simply not aware (the CLA, for example, claims to be able to reliably measure ‘value added’ learning that is beyond what is predicted by ability at enrollment, but seems to be curiously mum about the year-to-year stability of scores).  Anyways: I don’t have any easy answers, but I’m going to keep digging.

References:
Pascarella, E. T., & Terenzini, P. T. (2005). How college affects students: Vol. 2. A third decade of research. San Francisco: Jossey-Bass