The Serials Evaluator's Algorithms and the UIUC Data
The Serials Evaluator offers two algorithms for determining which serials should either be canceled or not purchased. One algorithm fully exploits any divergence of cost from ST value by selecting for cancellation or nonpurchase any title whose percentage of total cost exceeds its percentage of total ST value. The other algorithm allows the user to set goals in terms of cost reduction and ST value retention. In this algorithm the Evaluator forms two different sets—one from the titles with the highest prices, another from the titles with the highest ST value—and then compares these two sets to select for cancellation or nonpurchase only those high-price titles that are not in the high-value set. The latter option is more conservative, because it establishes controls only at the extremes of the distributions while allowing for random error at the lower ranges. This was deemed important because any set of library data contains considerable random error—particularly at the lower ranges of the distributions where the interval distances are extremely close. It was decided to utilize the second algorithm for the purposes of the exercise.
Investigation of the effect of employing this algorithm with all three of the Evaluator's measures of ST value—expert ratings, total ISI citations, and library use—on actual library use as well as on the costs of this use was made possible when the UIUC Chemistry librarian supplied us with data from the three use studies that she had done in 1988, 1993, and 1996 (Chrzastowski 1991; Chrzastowski and Olesko 1997). The 1993 UIUC Chemistry Library study (conducted from January 4 through March 31, 1993) was contemporaneous with the SRP pilot project of April 1993. A database of chemistry journals had been constructed as a result of the SRP pilot project that contained ratings by the LSU chemistry faculty, 1993 SCI citation measures and prices, as well as their publisher type and country of publication (Bensman 1996).
There were 120 titles costing $160,226.10 that were common to both the UIUC and LSU sets. UIUC use was compiled by having student workers count titles as journals were reshelved, returned from a two-hour loan period, or circulated via interlibrary loan. Both bound and unbound use was counted, and use was classified by date of journal publication into the following temporal categories: pre-1960, 1960s, 1970s, 1980s, and 1990s. For purposes of our investigation, UIUC use (which had been collected over a three-month period) was annualized by multiplying it by four.
Faculty Score, Citations, and the Sociometric Concept of Library Use
Acquisition of the UIUC data compelled a rethinking of the ST value measures utilized in the Evaluator's algorithms—especially with respect to library use. These measures had been conceptualized as sociometric ones, and they were based upon philosophic idealism, particularly Bishop Berkeley's contention that the essence of an object—in this case, ST value—is its being perceived. From this perspective, faculty ratings became the key measure, and total citations were accepted rather than impact factor, because total citations correlated better with faculty ratings by capturing the size as well as historical and social significance of the serials (Bensman 1996, 156–59). The good correlations of LSU faculty ratings with total citations were corroborated with 1994–95 SRP survey data in the test curriculum cores of Agronomy, Chemistry, and Mechanical Engineering, as shown above. Given this orientation, the equivalent sociometric measure in terms of library use would be total use over the entire backfile of serials.
The first stage of the investigation of the effect of the Evaluator's second algorithm on library use and the costs of this use was done on the sociometric basis, i.e., with LSU faculty ratings, total citations, and total UIUC Chemistry Library use. Analyses conducted with UnCover data had shown that high LSU faculty ratings and high ISI citations were strongly associated with high library use. However, the UnCover analyses lacked set definitions and utilized the flawed citation measure of impact factor. This had necessitated the utilization of nonparametric statistics within broad categories to neutralize the effect of outliers. However, the happy coincidence of use data from the UIUC Chemistry Library together with LSU faculty ratings and total citation measures in the same subject set and in the same time period enabled the utilization of more powerful parametric statistical techniques.
As usual, the method of analysis was to compute the Pearson product-moment correlation coefficient between the variables in question, regress one variable on the other to determine the outliers, and then recompute the Pearson product-moment correlation with the outliers excluded. With respect to faculty score, the initial correlation between it and total UIUC use was 0.73. To find the outliers, faculty score was regressed on total UIUC use, and analysis of the residuals indicated 3 outliers. In all three cases, actual total UIUC use was far lower than that predicted by LSU faculty ratings. These outliers shared the same characteristics of being at the extreme lower end of the total UIUC use distribution and having a relatively narrow subject focus. Therefore the cause of their being outliers may have been either a lower interest in such subjects at Illinois than at LSU or a sporadic use pattern not fully captured by the three-month survey. Their exclusion raised the correlation between LSU faculty ratings and total UIUC use to 0.75.
The results were even better with total ISI citations, which had an initial correlation of 0.82 with total UIUC use. Four outliers were pinpointed by the regression of total ISI citations on UIUC use. With three of these, the actual use was again far below that predicted by ISI total citations, and two had also appeared as outliers with LSU faculty ratings. The third such outlier was of a similar nature in that it also was located at the extreme lower end of the UIUC use distribution and had a narrow subject focus. Much different was the fourth outlier, Chemical and Engineering News, which had an observed use far above its predicted use—792 to 55. This result was natural, because citations are a measure of research quality, whereas the purpose of Chemical and Engineering News is to serve as the chief informational publication of the American Chemical Society. With the exclusion of these outliers, the correlation between UIUC use and total ISI citations rose to 0.86. For comparative purposes, the initial correlation of total UIUC use with impact factor was only 0.36—a natural result, because impact factor is controlled for size and time whereas the library use was not.
The Pearson product-moment correlations of total UIUC use with LSU chemistry faculty ratings and total ISI citations were extremely high. Calculations of the coefficients of determination showed that 56.3% in total UIUC use was explained by LSU faculty ratings and 74.0% by total ISI citations. Two major conclusions can be drawn from these high correlations. First, when one controls for outliers by defining proper sets, there will emerge high correlations between library use on the one hand, and either expert ratings or total citations on the other. Second, the ability of LSU chemistry faculty ratings to predict so accurately total UIUC Chemistry Library use demonstrates that university chemistry departments belong to the same social stratification system and require basically the same set of journals.
The three sociometric measures of ST value were plotted against price. Given the high correlations, it was not surprising that all three plots revealed the same bifurcated pattern found previously by Bensman (1996, 166–67) with ST value concentrating on the serials of the U.S. associations and costs concentrating on the serials of the commercial, largely foreign, publishers.