Type and Nature of ST Value Measures
The Serials Evaluator was designed to utilize three different measures of scholarly and ST value: expert ratings, ISI citations, and library use. Each of these measures has its strengths and weaknesses, and total reliance should not be put on any one of them. Of these measures, the one under investigation, expert ratings, is politically the most sensitive one on a university campus, where the expert ratings are done by the faculty. This political problem has been pithily summed up by Sapp and Watson (1989, 286) in the following passage:
Theoretically, faculty ratings should be a good measure of value. It is a global one, because the faculty should be able to take into account all aspects of a given serial—its value in research, instruction, and for general information—in assigning it a rating. However, in practice it has been found that the faculty tend to emphasize the research aspect of serials, and librarians have been forced to take measures to protect general periodicals and those more suitable for use by undergraduate students (Joswick and Stierman 1995; Sapp and Watson 1989, 287). The problem of underemphasizing general and undergraduate periodicals was not encountered in the SRP due to the nature of the LC class groups under analysis. Nevertheless, major flaws requiring corrective action were discovered in faculty ratings even from the research point of view, and they will be discussed below.
An interesting feature of faculty ratings is that there appears to be a high degree of consensus among faculty members in the same discipline but at different universities and colleges about the relative importance of individual serials. This was found by Goehner (1984a; 1984b) in a survey of 178 faculty members at 26 different institutions in 6 disciplines. Such a finding should not be surprising, because faculty members function within the social structure of their disciplines, and this is one of the main reasons why the LSU chemistry faculty gave such high ratings to the journals of the American Chemical Society. In a survey conducted at California State University at Dominguez Hills, Broude (1978, 163) found that the public administration faculty there also highly valued association journals and suggested none of these for cancellation.
In contrast to expert ratings, ISI citations have a more restricted applicability, because in effect they primarily measure the importance of a journal for research but not for instruction or general information. Moreover, even in terms of research, ISI citations cannot be utilized as a measure of value in certain cases. For example, citations have not been found to be applicable in the humanities. Despite plans to the contrary, ISI has not developed a JCR for the Arts and Humanities Citation Index (Garfield 1980, 55), and citation-based measures were not employed in either the 1981 or the 1993 assessments of U.S. research-doctorate programs in the humanities, while they were ultimately used for all other fields. Nevertheless, for those disciplines where citations have become an established practice, citations are an excellent measure of research quality as well as of the importance of serials in the overall social structure of their respective disciplines.
A complicating factor in the utilization of JCR citation data is the distinction between total citations and impact factor. Total citations can be roughly defined as the total number of citations received in a given year by a journal, including its entire historical backfile. On the other hand, impact factor was developed at ISI in 1973 to create a normalized measure of value by controlling the citation frequency of a serial for age and size for the purpose of comparing small journals to large journals as well as to compare journals within small or large research disciplines (Garfield 1997). This is done by limiting the backfile of a serial to the two years preceding the processing year of the JCR and then dividing the references given during the processing year to this two-year backfile by the number of source items in this backfile. This creates an average citation rate per citable unit.
In the analysis of the survey data gathered by the 1993 pilot project with the LSU Department of Chemistry (Bensman 1996), a good correlation—0.72—was found between faculty ratings of serials and their total citation rates. However, the correlation between faculty ratings and impact factor was only 0.27, and two basic reasons were found for this: (1) impact factor is distorted by the higher average citation rates of review articles, and (2) faculty ratings are heavily influenced by size—one of the very things for which impact factor controls. The utility of impact factor was further diminished by the finding that size is one of the major elements that determines the price of serials, and therefore direct comparisons between impact factor and price cannot be made. Further consideration also led to the conclusion that impact factor cannot be utilized in precise comparisons with library use because logically the latter also has to be influenced by size. However, as will be shown below, once one is aware of the basic characteristics of impact factor, one is able to design statistical techniques capable of properly using it as a measurement of utility and value. Such techniques are of great practical benefit, because the JCRs rank journals within subject sets by impact factor only. Such a method of presentation is necessary, because impact factor is greatly affected by the average number of references in the citing papers and the average age of the papers being referenced—variables that differ considerably from discipline to discipline (Garfield 1997).
The final measure of value used by the Serials Evaluator is library use. In theory, library use should be the best measure of all, because it is the final expression of the operation of all the causal variables. However, library use suffers from two major handicaps. First, use data are notoriously difficult to collect, especially for use that takes place within the library and cannot be captured by any of the monitoring systems. Second, there may be a great deal of random error in library use, particularly at institutions such as LSU with large undergraduate populations. This is evident in the inability of Ravichandra Rao to fit the NBD to undergraduate use data as well as in the finding by Metz (1983, 81) that knowledge of an undergraduate's major was significantly less predictive of the library materials the undergraduate would borrow than knowing the departmental affiliation of a faculty or graduate student borrower.