From: Sandy Thatcher <[log in to unmask]>
Date: Thu, 20 Dec 2012 10:49:38 -0600

Does anyone ever conduct formal reviews of journals?  I remember that
The Times Higher Education Supplement in the UK performed this
valuable service once upon a time (because one of our journals at Penn
State received a favorable review there), but I'm not sure whether it
still does, or whether this goes on anywhere else.

Some journal publishers themselves, as a matter of internal quality
control, may conduct reviews of their own journals from time to time.
I believe that a number of university presses do so, for their
journals and also monograph series, and report the findings to their
faculty editorial boards. We had this kind of procedure in place at
Penn State Press when I was director.

Such a review would normally look retrospectively at what was
published in the journal or series for five or more years and seek
external reviews from a range of experts in the field just as is done
for reviews of individual articles or monographs, but with a larger
number of reviewers involved, more like what is done for a tenure or
promotion review. One point in time at which this might be
appropriately done is when a journal editor's contract comes up for
renewal.

This kind of review would be very effective in further weeding out
"predatory" journals, and one could place more confidence in this kind
of formal review than in the informal reviews now being done. I
realize, of course, that given the number of journals, it would be a
daunting task, but perhaps if it were done systematically by field,
with each scholarly association responsible for quality control in its
own field (though some other body would need to review the journals
published by the society itself), this could be a sort of
self-policing exercise that would be appropriate for every society to
undertake as one of its major responsibilities.

Sandy Thatcher



At 11:13 PM -0500 12/19/12, LIBLICENSE wrote:

> From: Ken Masters <[log in to unmask]>
> Date: Wed, 19 Dec 2012 10:23:45 +0400
>
> Hi All
>
> I agree that any publishers who exploit researchers' desires to be
> published and who charge for services that are not delivered should be
> exposed.  (Naturally, this should apply to open access and non open
> access publishers alike).
>
> But I believe that the process should be formalised properly,
> otherwise it devolves quickly into what may look like a screaming
> match of accusations and counter-accusations.  Before any exposing can
> be done, the criteria should be clear, explicit, visible, relevant,
> and have strong support for being valid criteria.  (e.g. the fact that
> a publisher is large or small, is seldom (ever?) relevant.  The fact
> that a publisher is based in a "foreign" country, is not only
> irrelevant, but smacks of the worst kind of jingoism and even racism,
> and I would certainly not put my name to a list that held that as a
> criterion.)
>
> Once the criteria have been established, for each publisher so named,
> there should be explicit details and cases.  This should not be
> something based on hearsay or rumor or vague feelings of disquiet.
> The explicit documentation is crucial, and follows the same principle
> of any research - it allows any skeptics to examine the validity of
> the criticisms against the specific cases.  For example, simply saying
> that a publisher publishes material of questionable quality is
> meaningless, unless we have explicit information.  (E.g. to which
> articles are you referring? what are the inaccuracies? who performed
> the evaluation, and what qualifies them to perform such an
> evaluation?)
>
> It is unlikely that any one person is qualified (or has the time) to
> examine all the articles published by a particular publisher (unless
> the number of articles is very small), so making the claim to have
> done so immediately stains the credibility of the review unless that
> reviewer gives more details.  The reviewers would have to be explicit
> in their qualifications and background to justify the validity of
> their reviews - as much as I respect librarians, I do not know of many
> who have MDs (or the equivalent), and are therefore qualified (and
> have the time) to review a wide range of journal articles from a
> medical publisher.  If they do, then this needs to be stated to add
> credibility to the review.  The same would go for almost any other
> discipline.
>
> Along that line, one should also not confuse a publisher with a
> journal, or even a journal article - many respected publishers have
> had bad spots (e.g. to taint Elsevier because of a few shady editors
> is nonsensical, and will quickly make the critic look idiotic - to be
> balanced, we should take the same approach with all publishers).  We
> have all seen papers in respected journals that have questionable
> references, and we may wonder at how carefully the peer-reviewers have
> done their work - to taint an entire company as fraudulent on the
> strength of that is an abuse of a privileged position.  (And yes, I
> believe everybody on this list is in a very privileged position).  On
> this list, the number of retractions from respected commercial
> publishers has never led to their being labelled as "predatory," yet,
> if the retraction is to be taken as admission of error, then it is
> obvious that those articles were of questionable quality, and the
> publishers made money from them.
>
> The value of having this credibility would allow one to go one stage
> further: checking on the review process.  For example, if a paper were
> questionable, then the editor of the journal could be mailed and could
> be asked to submit proof of peer-review (No demands, no accusations,
> requests).  The editor would need to check that this was ok with the
> author and the reviewers, and there may be issues of confidentiality,
> but I would hazard that it may be a painless process. After a proper
> evaluation of the review documentation, one could determine the cause
> of the problem: maybe the reviewers just messed up; maybe an author
> was 'bullied' into writing something s/he didn't want to write (yes,
> it happens), etc.
>
> And when the review of the publisher has been written, then follow
> your own criteria to the letter.  Don't self-publish it in a blog that
> has no peer-review, and over which your have the facility to quash
> dissenting voices by not publishing their comments.  Rather, submit it
> to a respected journal with proper peer-review, and peer-reviewers who
> are committed to pick up on any unfounded generalizations and
> speculations, and have the paper go through the process that we demand
> from respected publishers.  (This appears self-evident, given the
> subject.)
>
> Yes, this does involve work.  Without it, however, the classification
> or criticisms have no validity at all.  When we reach that stage, then
> the criticisms become valid criticisms of a system or publisher.  And,
> in so doing, we contribute to the quality of research everywhere.
>
> Disclaimer: I am the editor of an open access journal whose publisher
> appears on a list of "predatory" publishers.   It's no secret that
> Jeffrey Beall and I have a difference of opinion on several issues
> (which I believe could be amicably resolved), but I don't agree with
> the tactics that he has reported have been used against him.
>
> Regards
>
> Ken
>
> Dr. Ken Masters
> Asst. Professor: Medical Informatics
> Medical Education Unit
> College of Medicine & Health Sciences
> Sultan Qaboos University
> Sultanate of Oman
> E-i-C: The Internet Journal of Medical Education