This timely conversation drew a big interdisciplinary crowd from all around campus—from faculty, to librarians, to undergraduate students—interested in how the new university uses data and how that affects higher education in our contemporary moment. Our six presenters gave short presentations to open the floor up for a larger discussion.
Our first speaker, Brian Cameron, who is the head of Library Collection Services at Ryerson University in Toronto, presented on the state of impact metrics from the libraries’ point of view. His presence was very much welcomed by the Welch and Sheridan Libraries staff who were in attendance. Cameron mostly focused on the uses—and especially misuses—of impact metrics in tenure and faculty promotion cases. Many administrators across universities in North America, Cameron suggested, seek objective metrics to measure faculty productivity; they find the alternative process of comprehensive qualitative analysis too burdensome. But metrics like the journal impact macros lends themselves all too easily to willful manipulation. The journal impact factor was created in the 1960s as a metric to help decision-making regarding which journals should be covered in the Web of Science. Libraries used it for collection purposes and authors used it to determine which journal to publish in, both of which can be said to be a misuse of the metric—a dilution of its modest objective. today, 129 universities around North America use the impact factor to evaluate the quality of their faculty’s research profiles. The massive problems associated with this kind of use include how, in some disciplines, the number of citations and the quality of research scarcely correlate, or how the measuring of average citations has, in the recent past, skewed journals’ interest overwhelmingly towards publishing review articles, which usually get more citations, and discarding case studies. Cameron identified three categories of problems with the current state of bibliometrics. One is manipulation, as in the cases of self-citation, or questionable practices such as journals extending the life of articles online before print publication. (The metric takes into consideration the citations that an article has received in its first two years after print publication; stretching an article’s online life will inflate the citation number going into the metric.) Another category of issues is proliferation. Using the same datasets, many new metrics are in circulation mimicking the impact factor (like the “science” impact factor) and bordering on scamming. Academic analytics are introduced into the research environment as cost-saving measures, which then contribute to the neglect of teaching quality in faculty evaluation. The final and related category of problems is the metrics’ commercialization, giving the metrics builders the profit incentive to continue colonizing higher education.
Ian Lowrie is an anthropologist who studies Russian data scientists in the education context. After receiving his Ph.D. from Rice University and lecturing at Portland State University and Lewis and Clark College, Lowrie joined Ad Hoc, a civic technology company dedicated to improving the federal government’s data processing infrastructure. His presentation on the state of data science in contemporary Russia gave the event a much needed counterpoint to think through our instinctive suspicions of the discipline. In the case study he presented, Russian data scientists use their science to seek a rationalized form of governance free from the suffocating conformity in Russian educational system. In Russia, data scientists seek these metrics as a way to to weed out outdated curriculum and non-productive science; precarious faculty welcome the promise of reducing corruption and caprice in decision-making about advancement and salary. For these professionals, the move towards data science-powered metrics is a calculated political move in the hopes for a better career path for themselves. Many of Lowrie’s interlocutors stayed in the university despite terribly low wages as a result of their commitment to science as vocation; they could easily choose to fail “up” by leaving academia for industry. The techniques we consider as neoliberal were not foisted on them; there is an argument to be made, Lowrie pointed out, against perceiving these people as either pernicious neoliberals or unwilling dupes participating in an audit culture.
Nicholas Hartlep, the Robert Charles Billings Chair in Education at Berea College, gave a talk centered on the question of what it means to live by your morals in the university environment today. The talk was particularly informed by Hartlep’s experience as the director of an open access journal and faculty at a college that has the moral mission of educating only low-income students. Hartlep gave examples of platforms that can be used by scholars to share their work—platforms that seek to undermine the equation of reading and sharing scholarship freely with theft under current law. (An example is the website: https://www.howcanishareit.com/) Hartlep also dwelled on what he called the trap of meritocracy, where faculty are drawn to activities that are not necessarily scholarly but curate the appearance of productivity, such as managing their social media.
Shreeharsh Kelkar, a lecturer in Interdisciplinary Studies at UC Berkeley, studies the making of MOOCs (Massive Open Online Courses) as educational platforms created and directed by data scientists. His presentation was based on his multi-year ethnographic fieldwork at edX one of the pioneering MOOC producers in the country. Throughout this study, Kelkar explored how data science interfaces with substantive expertise (or domain expertise), ultimately finding that data science has acquired the authority to define what substantive expertise is and why it matters—a kind of authority that explains the currency of data science in our society today and why we might want to think critically about it. A key example Kelkar gave was on the discussion around the use of videos in MOOCs. Based on their studies on what learners respond to best, data scientists at edX have been pushing for 6 minute-long videos as the ideal format for course material. In this particular case, the data scientists backed their claim by pointing at how this claim has also resonated with video producers; it was a sleight of hand that subsumed video producers into the category of domain experts, while erasing their own authorship and normalizing the epistemology of data science itself. As can be expected, this claim was not received universally among educators, who pointed out its many flaws—how it generalized too much and collapsed the purposes of different kinds of videos onto each other. Nevertheless data scientists seemed to set the terms of how others may participate in the platform. Many a form of domain expertise upon which we previously relied is displaced this way. Kelkar ended by noting that as citizens of the university, we need to ask: who gets to ask the question that we then use data science to answer? Who decides what the relevant domain expertise is and who the domain experts are?
Chris Morphew, dean of the JHU School of Education, started his talk titled, “On the Efficacy of Rankings” by noting his own participation in the techniques discussed so far—by noting that he might, in fact, count, as one of the “neoliberal managers” that our presenters have referred to pejoratively. There is a chance, he remarked, that the university has strayed from its missions, which is what led to the creation of entire industries around us; it is perhaps because “we didn’t do our job.” One metric—an admittedly flawed one—is university rankings; but what does their growth tell us about what is going on in universities? How might leaders navigate them given their ubiquity and ever-expanding functions? Here again, a bit of history as to where the metric came from is useful to remember. The US News rankings began in 1983 to sustain a dying industry—newspapers— and at first widely dismissed by established institutions like Harvard. Their promise was that they could create order in a chaotic, hard-to-understand industry that is higher education; they could signal quality, sort students and resources. Morphew remarked that the metric caught on, because “We have done a bad job explaining to consumers what our product is” (and noted right away that he is aware that the terminology of “consumer” and “product” is not always welcomed by audiences). Rankings are partly a result of the failure of higher education to be transparent; we, in fact, have a lot of data on what students learn, how they benefit from what they learn, their socioeconomic paths, what paths they might choose after graduation, how many students are finishing graduate school—none of which are used as part of the calculation for rankings. A productive idea going forward would be to cultivate other means of sharing information and focus on what really matters: what do rankings measure that we want to improve at our own universities?
Our final speaker was our very own Veena Das. Das responded to the discussion points on the floor in three ways, particularly focusing on how the metrics affect and transform research. The first question concerned what counts as knowledge—what kind of shift in our understanding of knowledge occurs when our everyday research practices absorb metrics like rankings and the impact factor. A second question concerned how scientific authority is claimed. The pressures of performing well in university rankings produce changes in how one could make research cited for policy purposes by universities and by others who take the university to be the producer of authoritative knowledge. A final question was whether inequalities in knowledge would go up or down with the rise of metrics. We know, for instance, that one thousand African universities applied to be covered in US News rankings and only three were able to get in, despite the absence of a comprehensive study of knowledge production and the state of research at those institutions. The metrics are creating an interesting geography of knowledge, which, for instance, can discard the experiences of huge populations (in the example Das gave, millions of tuberculosis patients in India) for being too “specialized” an audience. This codes generalizability as the character to align with the forms that circulate in the US; it deems work that is coming out of everywhere else as “non-theoretical” case studies. There is an imminent conceptual problem here of knowing from the outset what the outcome of research must be. Another example Das discussed was close to home—a JHU-stamped white paper that argued that the presence of campus police reduces crime. Upon close analysis, Das and her team of research assistants found that, of the 79 articles that were cited in the paper, perhaps only four had any connection to the subject at hand (and even then did not argue definitively in the way in which they were cited). This situation casts a fundamental doubt on the success stories coming from universities’ self-studies. The universities, it turns out, are very good at hiding their own failures.
Our lively question and answer session was moderated by Professor of Classics, Shane Butler. The first line of questioning concerned the audience members’ struggle with the idea of quantitative metrics playing too large a role in promotion decisions that are so central to departmental quality, not to mention individual faculty’s professional and personal wellbeing. Several participants highlighted the difficulties of measuring critical thinking. Dean Chris Morphew pushed back on a mere reliance on qualitative assessments by bringing up the value that metrics provided by third party firms can bring to the table—a metric, for instance, that compares individual faculty and departments to their peers elsewhere, and gauges the value of individual faculty to the university (and, by extension, the value that would be lost by losing that faculty). Sitting uncomfortably with this suggestion, participants pondered the pedagogical models that would not treat students as customers. Others brought up different kinds of issues that trickle down from bibliometrics, like the emergence of English at the center of an enforced, neocolonial monolingualism in the publication environment. Das pointed out that, as anthropologists would readily tell you, microgeographies of power persist in the era of metrics, despite what the burden of civility in the US would have you believe. Perhaps, Shane Butler pointed out, there is simply too much assessment going around.
We also had a productive discussion around the need for nuance—an ability to tell the different kinds of harm and benefit accruing in the data environment. The librarians in the audience discussed with Cameron the difficulties of communicating such nuances to faculty—like the difference between publication platforms like Elsevier and SciHub (which was sued by Elsevier) which might all too easily be collapsed together. There is constant work that needs to be done to exist with responsibility and productivity in this environment—to promote open access, to remember to get an Orcid identifier, to consider, in the case of graduate students, how to publish parts of dissertation research. Lowrie also made a case for the need for nuance—to be able to tell the difference between data scientists steeped in a culture of metrics and oppressive higher management. Data scientists, Lowrie pointed out, would simply tell you that the US News rankings was bad data science or “statistical gerrymandering.” Responding to Kelkar’s example of data scientists and video producers, Lowrie pointed out that his informants would probably find that instance to be simply bad data science for failing to tie the data back to educational outcomes. There is value, in other words, in distinguishing what the producers of metrics are trying to accomplish and what the management is. If you are in a room with administrators and data scientists, he suggested, there is a higher chance that data scientists will listen to you. A powerful closing question came from our own Naveeda Khan. What is the failure that the university is trying to respond to? Who framed that failure? The answers remain open.