"Refereed Journals: The Emperor’s New Clothes"

Refereed Journals: The Emperor's New Clothes

by Antonio J. García

So many university administrators and committees place such emphasis on their faculty members publishing in “refereed” or “peer-reviewed” journals when, in fact, no standard exists for such a periodical. This fallacy represents the height of “The Emperor’s New Clothes” in academia. Many higher-education institutions line the streets applauding blindly as this naked monarch of tradition processes before us year after year. Relatively few individuals seem aware that the promotion and tenure process, which typically leans so heavily on an evaluation of one’s research and external publication, is so often based on this house of cards termed “refereed journals.” The topic comes to mind as I prepare yet another in an endless stream of external reviews for professors seeking tenure or promotion at other institutions.

I first learned of the scope of this contradiction in 1999, when researching how to obtain refereed-journal status for a publication for which I was serving as editor. I contacted the best-known listing of journals and other publications, Ulrich’s Directory of Periodicals, and sought the standard by which such lines could be drawn. Joyce Watson, then the Supervisor of R.R. Bowker, promptly replied on May 27, 1999 that as to periodicals, “the determination of whether they are peer-reviewed or not is done before we get them.” In other words, the periodicals’ editors decide if their periodicals are refereed or not.

I spent the next several days contacting the editors of journals spanning engineering, music, and more. “Sorry, I can’t help you with that one,” wrote one editor. “I think ‘refereed’ is a loose and ill-defined term and doesn’t mean very much,” responded another. “It’s true, our periodical describes itself in the frontmatter as a ‘refereed journal,’” replied a third; “but I don’t know who originated this statement or by what authority. We don’t even publish our list of reviewing evaluators. A neighboring journal, refereed in the same sense, has an Editorial Board that is advisory to the editor and meets once a year for an hour of general discussion. It didn’t occur to me that there is some threshold that must be crossed for a journal to be ‘refereed.’ My impression is that the status of ‘refereed’ is largely self-defined by each journal. I don’t think we ever submitted a request to some board to acquire this status, and in any case we don’t report to anyone.”

The latter respondent referred me to the American Council of Learned Societies, of which his association and the other he had referred to were members. Candace Frede, ACLS Director of Publications at the time, replied to my request: “I directed your inquiry to my colleagues here at ACLS, but we do not know of any definitive source of information for what constitutes a refereed journal.” Indeed, the Guidelines for Contributors to the Council of Editors of Learned Journals at the time stated, “Most professional publications referee their contents. Submissions invited by guest editors or regular editors for special or regular issues may or may not be counted as refereed articles; contributors should check with their home campus about such policies.”

So, no luck with the “Learned Societies” and “Learned Journals” in defining what makes such a learned periodical!

Many home campuses have attempted to figure out what should count as a refereed article or journal, with wide variances in results. At the time, my quest uncovered a page at the Colorado State University Libraries that had been created by Linda Shackle (Arizona State University Librarian, Noble Science and Engineering Library, Science Reference Department) along with colleague Naomi Lederer of CSU Libraries.

This document offered “General Criteria that can be used to distinguish between popular magazines, trade magazines, and scholarly journals.” These criteria included whether the periodical’s appearance included an “eye-catching cover,” “glossy paper,” and “heavy advertisements” versus “plain cover,” “plain paper,” and “ few or no advertisements.”

Yet they were quick to point out: “Some journals do not meet all the criteria in one category. For example, ‘Scientific American,’ which has glossy pages and color pictures, contains both scholarly articles as well as those geared to a more general audience. Accountability and content of the specific article are the key criteria used to determine if an article is scholarly.”

Imagine that: “Accountability and content of the specific article are the key criteria used to determine if an article is scholarly.” A voice of reason in the desert! Furthermore, they continued, “Ulrich’s Directory of Periodicals...now has a list of refereed journals. However, not all scholarly journals are on this list.” So, we have a scenario in which the best-known source for listing refereed journals cannot define that standard, nor is it a comprehensive listing—nor would such a comprehensive listing ever be possible!

On June 3, 1999 I phoned Ulrich’s, gave my name and my periodical’s, and asked that it be listed as refereed. My request was granted instantly: no documents shared, no background check, not even identification shown. Done. It was far easier than obtaining a subscription to our periodical and took under a minute on the phone.

Not much has changed since 1999. In 2016, Ulrich’s online FAQ page stated, “As used in the Ulrich's knowledgebase, the term refereed is applied to a journal that has been peer-reviewed. Refereed serials include articles that have been reviewed by experts and respected researchers in specific fields of study including the sciences, technology, the social sciences, and arts and humanities.... The Ulrich's editorial team assigns the ‘refereed’ status to a journal that is designated by its publisher as a refereed or peer-reviewed journal. Often, this designation comes to us in electronic data feeds from publishers. In other cases Ulrich's editors phone publishers directly for this information, or research the journal’s information posted on the publisher’s website.” There remains no clear standard.

As I confirmed also via an e-mail exchange with Ulrich’s in 2008, the periodicals’ editors still call the shots. “You are correct,” stated Christopher King: “Ulrich’s does not specifically evaluate/determine whether or not a journal is peer-reviewed and relies on the publisher/editor of a journal to provide us with the designation. We do have certain basic criteria for what constitutes a refereed/peer-reviewed serial (reviewed by a group of specially chosen professionals using a manuscript peer review and evaluation system which attempts to protect the quality of scholarly material published) and will add/remove the designation if we feel we have sufficient information to do so.”

As a parallel, allow me to offer a comparison with another excellent service-listing, that of copyrights, as handled by the Library of Congress. Most applicants for copyright believe that once you’ve received the reviewed copyright form from the Library of Congress, embossed with the official, raised seal, you own your work, duly recognized by the government. In fact, the document serves only as the equivalent of a rubber-stamped time-clock: the government assumes you had the right to claim copyright in the work and will operate on that assumption until someone else might step up to assert the contrary. (And, in fact, you already own your work even before you file the paper with Washington.) Similarly, Ulrich’s assumes a periodical has the right to claim it is peer-reviewed and provides said listing until such time as evidence arises to its attention to the contrary. And exactly who is going to place a phone call to Ulrich’s to report that a listed refereed journal is in fact not refereed—especially when no clear standard exists?

My concern is not at all with Ulrich’s (any more than I have qualms about the Library of Congress). It provides an excellent service by listing periodicals, and it states clearly that it cannot judge which are in fact refereed; it merely takes the word of the periodicals. I also have no qualms about the notion of a refereed journal and actually edited what I believed to be a fine one years ago. I furthermore believe that non-refereed periodicals often publish articles generating as much or more impact on the educational community and the world at large and therefore should not be summarily and categorically dismissed or devalued.

My concern is with those university administrators and/or committees who invest an interpretation of “refereed journal” that simply does not exist: there is no standard—you can have your own, homegrown periodical listed in Ulrich’s as “peer-reviewed” without much effort.

What should be the key factor of quality is the “accountability and content of the specific article,” not the surrounding binding of its pages or whether its periodical is notated in Ulrich’s as refereed. The sooner every university administrator and tenure/promotion committee member wakes up and smells the magazine with his or her coffee, the sooner more equitable estimations of research output within tenure/promotion reviews will be the norm.

It’s been almost twenty years since I had discovered this truth about The Emperor’s New Clothes in academia. I hope it won’t take another decade or more before the majority of university faculty and administrators understand it as well.

Antonio J. García is a Professor Emeritus and former Director of Jazz Studies at Virginia Commonwealth University, where he directed the Jazz Orchestra I; instructed Applied Jazz Trombone, Small Jazz Ensemble, Jazz Pedagogy, Music Industry, and various jazz courses; founded a B.A. Music Business Emphasis (for which he initially served as Coordinator); and directed the Greater Richmond High School Jazz Band. An alumnus of the Eastman School of Music and of Loyola University of the South, he has received commissions for jazz, symphonic, chamber, film, and solo works—instrumental and vocal—including grants from Meet The Composer, The Commission Project, The Thelonious Monk Institute, and regional arts councils. His music has aired internationally and has been performed by such artists as Sheila Jordan, Arturo Sandoval, Jim Pugh, Denis DiBlasio, James Moody, and Nick Brignola. Composition/arrangement honors include IAJE (jazz band), ASCAP (orchestral), and Billboard Magazine (pop songwriting). His works have been published by Kjos Music, Hal Leonard, Kendor Music, Doug Beach Music, ejazzlines, Walrus, UNC Jazz Press, Three-Two Music Publications, Potenza Music, and his own garciamusic.com, with five recorded on CDs by Rob Parton’s JazzTech Big Band (Sea Breeze and ROPA JAZZ). His scores for independent films have screened across the U.S. and in Italy, Macedonia, Uganda, Australia, Colombia, India, Germany, Brazil, Hong Kong, Mexico, Israel, Taiwan, and the United Kingdom. He has fundraised $5.5 million in external gift pledges for the VCU Jazz Program, with hundreds of thousands of dollars already in hand.

A Bach/Selmer trombone clinician, Mr. García serves as the jazz clinician for The Conn-Selmer Institute. He has freelanced as trombonist, bass trombonist, or pianist with over 70 nationally renowned artists, including Ella Fitzgerald, George Shearing, Mel Tormé, Doc Severinsen, Louie Bellson, Dave Brubeck, and Phil Collins—and has performed at the Montreux, Nice, North Sea, Pori (Finland), New Orleans, and Chicago Jazz Festivals. He has produced recordings or broadcasts of such artists as Wynton Marsalis, Jim Pugh, Dave Taylor, Susannah McCorkle, Sir Roland Hanna, and the JazzTech Big Band and is the bass trombonist on Phil Collins’ CD “A Hot Night in Paris” (Atlantic) and DVD “Phil Collins: Finally...The First Farewell Tour” (Warner Music). An avid scat-singer, he has performed vocally with jazz bands, jazz choirs, and computer-generated sounds. He is also a member of the National Academy of Recording Arts & Sciences (NARAS). A New Orleans native, he also performed there with such local artists as Pete Fountain, Ronnie Kole, Irma Thomas, and Al Hirt.

Mr. García is a Research Faculty member at The University of KwaZulu-Natal (Durban, South Africa) and the Associate Jazz Editor of the International Trombone Association Journal. He has served as a Network Expert (for Improvisation Materials), President’s Advisory Council member, and Editorial Advisory Board member for the Jazz Education Network . His newest book, Jazz Improvisation: Practical Approaches to Grading (Meredith Music), explores avenues for creating structures that correspond to course objectives. His book Cutting the Changes: Jazz Improvisation via Key Centers (Kjos Music) offers musicians of all ages the opportunity to improvise over standard tunes using just their major scales. He is Co-Editor and Contributing Author of Teaching Jazz: A Course of Study (published by NAfME), authored a chapter within Rehearsing The Jazz Band and The Jazzer’s Cookbook (published by Meredith Music), and contributed to Peter Erskine and Dave Black’s The Musician's Lifeline (Alfred). Within the International Association for Jazz Education he served as Editor of the Jazz Education Journal, President of IAJE-IL, International Co-Chair for Curriculum and for Vocal/Instrumental Integration, and Chicago Host Coordinator for the 1997 Conference. He served on the Illinois Coalition for Music Education coordinating committee, worked with the Illinois and Chicago Public Schools to develop standards for multi-cultural music education, and received a curricular grant from the Council for Basic Education. He has also served as Director of IMEA’s All-State Jazz Choir and Combo and of similar ensembles outside of Illinois. He is the only individual to have directed all three genres of Illinois All-State jazz ensembles—combo, vocal jazz choir, and big band—and is the recipient of the Illinois Music Educators Association’s 2001 Distinguished Service Award.

Regarding Jazz Improvisation: Practical Approaches to Grading, Darius Brubeck says, "How one grades turns out to be a contentious philosophical problem with a surprisingly wide spectrum of responses. García has produced a lucidly written, probing, analytical, and ultimately practical resource for professional jazz educators, replete with valuable ideas, advice, and copious references." Jamey Aebersold offers, "This book should be mandatory reading for all graduating music ed students." Janis Stockhouse states, "Groundbreaking. The comprehensive amount of material García has gathered from leaders in jazz education is impressive in itself. Plus, the veteran educator then presents his own synthesis of the material into a method of teaching and evaluating jazz improvisation that is fresh, practical, and inspiring!" And Dr. Ron McCurdy suggests, "This method will aid in the quality of teaching and learning of jazz improvisation worldwide."

About Cutting the Changes, saxophonist David Liebman states, “This book is perfect for the beginning to intermediate improviser who may be daunted by the multitude of chord changes found in most standard material. Here is a path through the technical chord-change jungle.” Says vocalist Sunny Wilkinson, “The concept is simple, the explanation detailed, the rewards immediate. It’s very singer-friendly.” Adds jazz-education legend Jamey Aebersold, “Tony’s wealth of jazz knowledge allows you to understand and apply his concepts without having to know a lot of theory and harmony. Cutting the Changes allows music educators to present jazz improvisation to many students who would normally be scared of trying.”

Of his jazz curricular work, Standard of Excellence states: “Antonio García has developed a series of Scope and Sequence of Instruction charts to provide a structure that will ensure academic integrity in jazz education.” Wynton Marsalis emphasizes: “Eight key categories meet the challenge of teaching what is historically an oral and aural tradition. All are important ingredients in the recipe.” The Chicago Tribune has highlighted García’s “splendid solos...virtuosity and musicianship...ingenious scoring...shrewd arrangements...exotic orchestral colors, witty riffs, and gloriously uninhibited splashes of dissonance...translucent textures and elegant voicing” and cited him as “a nationally noted jazz artist/educator...one of the most prominent young music educators in the country.” Down Beat has recognized his “knowing solo work on trombone” and “first-class writing of special interest.” The Jazz Report has written about the “talented trombonist,” and Cadence noted his “hauntingly lovely” composing as well as CD production “recommended without any qualifications whatsoever.” Phil Collins has said simply, “He can be in my band whenever he wants.” García is also the subject of an extensive interview within Bonanza: Insights and Wisdom from Professional Jazz Trombonists (Advance Music), profiled along with such artists as Bill Watrous, Mike Davis, Bill Reichenbach, Wayne Andre, John Fedchock, Conrad Herwig, Steve Turre, Jim Pugh, and Ed Neumeister.

The Secretary of the Board of The Midwest Clinic and a past Advisory Board member of the Brubeck Institute, Mr. García has adjudicated festivals and presented clinics in Canada, Europe, Australia, The Middle East, and South Africa, including creativity workshops for Motorola, Inc.’s international management executives. The partnership he created between VCU Jazz and the Centre for Jazz and Popular Music at the University of KwaZulu-Natal merited the 2013 VCU Community Engagement Award for Research. He has served as adjudicator for the International Trombone Association’s Frank Rosolino, Carl Fontana, and Rath Jazz Trombone Scholarship competitions and the Kai Winding Jazz Trombone Ensemble competition and has been asked to serve on Arts Midwest’s “Midwest Jazz Masters” panel and the Virginia Commission for the Arts “Artist Fellowship in Music Composition” panel. He was published within the inaugural edition of Jazz Education in Research and Practice and has been repeatedly published in Down Beat; JAZZed; Jazz Improv; Music, Inc.; The International Musician; The Instrumentalist; and the journals of NAfME, IAJE, ITA, American Orff-Schulwerk Association, Percussive Arts Society, Arts Midwest, Illinois Music Educators Association, and Illinois Association of School Boards. Previous to VCU, he served as Associate Professor and Coordinator of Combos at Northwestern University, where he taught jazz and integrated arts, was Jazz Coordinator for the National High School Music Institute, and for four years directed the Vocal Jazz Ensemble. Formerly the Coordinator of Jazz Studies at Northern Illinois University, he was selected by students and faculty there as the recipient of a 1992 “Excellence in Undergraduate Teaching” award and nominated as its candidate for 1992 CASE “U.S. Professor of the Year” (one of 434 nationwide). He is recipient of the VCU School of the Arts’ 2015 Faculty Award of Excellence for his teaching, research, and service and in 2021 was inducted into the Conn-Selmer Institute Hall of Fame. Visit his web site at <www.garciamusic.com>.

| Top |