Yesterday, I hit a nerve on Twitter. Ok, more than one. But it resulted in a great discussion about open access and brought up some interesting questions. I’d like to take the opportunity to explain in more detail what I meant and did not mean by my tweet. And then I’d like to open up the discussion further. But first, the backstory.

I am writing a systematic review. For those not familiar with the concept, this is not simply summarizing work others have done in a particular area of research. It involves designing searches, implementing filters, and clearly outlining criteria for selecting or excluding articles. The idea is to give a complete overview of the literature and be able to quantify, for example, what percentage of studies in the research area used a certain technique, or arrived at a common conclusion.

To give you an idea of the scale of my review, my searches retrieved 370 results. 73 of these were duplicates and excluded, leaving me with 297 unique articles to screen. From these, I excluded 183 studies whose results were irrelevant to the questions my review seeks to answer, a further 8 which were review or comment articles, and another 7 studies which used techniques other than the ones to which I’m restricting my review. Many of the articles I retrieved and screened were published in subscription journals, but I gained access via a university affiliation. Even so, there were some articles I couldn’t retrieve. So, I tweeted this:

The responses soon came in.

I think Dennis and Matt are right. Papers shouldn’t be excluded due to non-scientific reasons, and systematic reviews should be complete. But I argue that my reason for excluding articles to which I don’t have access is scientific and related to completeness. First, from a practical standpoint, I can’t extract the information I need. My review involves specifying certain methodological details of each study – details which are not available in the abstracts. Second, and more generally, I cannot evaluate the quality or validity of a study if I can’t read it in full. There are other studies which I have excluded from the review after reading the full text and finding inconsistencies in the methods, or disagreement between statements in the methods and results sections. As a scientist, I think it is irresponsible of me to cite work I haven’t properly evaluated.

With respect to the review, my thinking was as follows. Typically, such reviews include a schematic with boxes enumerating at each stage how many articles were excluded and for what reasons. I planned to include a box for those 7 (I have now decreased that number to 5) articles that I could not access even with my university account. This seemed fair to me. Based on a lack of information, I couldn’t very well include them nor assign them to any other exclusion box, and I was clearly stating the reason for exclusion.

This is where the nerve was hit. Some suggested I was practicing “activism” rather than scholarship, “shunning paywalled literature” for the cause of open access. This is where I think my tweet was misunderstood. It’s no secret I am an advocate of open access, and perhaps I brought some of the misinterpretation on myself for having included the #openaccess hashtag in my tweet. But activism was not my intention here. I was not suggesting that people exclude articles from their reviews just because they are not open access. In fact, I am including at least 40 articles in my review which are paywalled. Unfortunately, because such a large proportion of biomedical research is currently paywalled, it would not be a very thorough or informative review if I excluded these articles. My intention was instead two-fold: (1) simply to point out that a scholar – and even one who does have access to a large portion of the paywalled literature – was still struggling to do a thorough review because of a lack of access, and (2) to alert (or remind) authors who publish in paywalled journals that their work may not get cited not because other researchers aren’t interested, but because they can’t read it.

Some tweeters responded that my job as a researcher doesn’t end when I hit an electronic paywall:

Many tweeted that I should contact the authors directly to request reprints or try interlibrary loans. I wondered whether, given that I have 67 other eligible studies I can read, is it worth the time and effort to track down the 7 (now 5) I can’t? In the interest of thoroughness, I suppose the answer is ‘yes’. But some questioned whether the burden was on me to do this:

While I wasn’t intending to argue that papers should be excluded just because they are paywalled, some were arguing exactly this point:

Others disagreed:

The debate hinged on communication. As a scientist, is it your responsibility to communicate your science? If you lock up your results, shouldn’t that be considered a failure of communication and therefore unscientific? Or, is it the responsibility of the person conducting the review to communicate effectively, including all relevant literature whether paywalled or not?

In fact, others suggested that including paywalled articles in open review papers could be seen as supporting open access by bringing those results out into the light:

People got passionate about this one. And I can understand why. I am passionate about open access. There are few things more frustrating as a researcher than seeing that nasty paywall come between you and science, except perhaps seeing that frustration in your students or your patients.

But they are ways you can actually hurt the cause you are trying to support. There are simple issues and complicated ones. Should researchers lock their work behind paywalls? No. I think that’s pretty simple. And if I’m expressing my personal views on the matter, I also think it’s unscientific. I express that few frequently with colleagues, collaborators. I write about it. I try to be as open as I can with my own work. That is my form of activism. However, given that a lot of relevant research is currently locked up, how should researchers like myself deal with that? That’s a little more complicated and I don’t pretend to have all the answers. I do believe we have to be careful that our views of what should be with respect to open access do not cause us to act unscientifically. Do I like that biomedical research is paywalled? No. Does it make me angry? Yes. Does that mean I have the right to exclude paywalled research from my review? I don’t think so.

So, I will make additional efforts to retrieve those articles I am missing. If I get them, great. If I don’t, then I can say my reason for excluding them was purely scientific. I could not evaluate the work. Period. And in the meantime, I’ll keep advocating for open access. Hopefully, the next time I do a systematic review access won’t be an issue.