On Systematic Reviews

I am performing a systematic literature review. 

This is not the first literature review I do (my first one on Music Generation Systems has even become relatively popular) but its the first systematic one I do. What does that mean? What is a systematic literature review? Let’s start with what it is not. A “standard” review (I’ve started to see the term “narrative” literature review) involves reading a lot of papers, ideally all the most relevant papers in a field, and then describing the general state of the field, what research questions have been posed, what solutions have been proposed, what worked and what not, and, most importantly, what future venues of investigation seem more promising. I personally believe this is the only thing that would qualify as a literature review, being indeed a review of existing literature.

A systematic literature review is not that. The crucial word here is “systematic”, a word that in this context takes a higher priority over the other two words. Wishing to be as “scientific” as possible, researchers have started to define ways to be as quantitative and repeatable as possible when doing reviews. The process, the method, is the most important part of the review, and justifies the inclusion or exclusion of papers. A typical process involves identifying research questions, formulating queries based on those, retrieving papers from academic databases using those queries, and selecting among the results only those papers that satisfy certain criteria (like publication date being recent enough, having passed through peer review, having at least n participants if trials are included, etc.). All and only the retrieved papers that satisfy those criteria are read for the review. Even the reading is systematic: instead of reading all the papers and writing down interesting facts emerging from the reading, as one would in an old-fashioned review, the scientist must have a form to fill in while reading the papers, to extract the information that ideally answers the original research questions. Finally, one can aggregate these forms and report the results in a paper. It is still possible to include general observations on the examined literature in the discussion, of course, but the main course is the results from the aggregated data extraction forms. In certain cases, one can even do meta-analyses on the extracted data (as is particularly useful and common in medical fields).

I believe it should be clear that the two things are rather different. So different that I believe the latter should be named differently from the first. While the first is a literature review, the second is more of a (systematic) literature querying: if one person has a questions, and wishes to (try to) find the answer to that question in the literature, this is the way to go.

On the other hand, if one person decides they want to give an overview on the state of the art on a certain field, or a certain sub-part of a field, I find this systematic approach to be a terrible idea. Why? Because, from my experience, a literature review becomes a very personal, living experience. You start reading some papers, and get a basic general idea on the main facets of the field. You check the references, and immediately find out some authors are more common than others, and maybe some papers are very popular. So you obviously want to read those. Or you do the opposite: you have further questions on a paper, and check the ones that have cited it later on to find if the authors went on with their work, or if others took a different spin on it, or again if someone pointed out a mistake or inaccuracy which you also spotted. Or sometimes, you read a paper and realize there is a whole different possible point of view on a topic, so you start digging in that direction. You change idea on what the topic of the review is. You broaden horizons. You ask further questions. You cannot do this with a systematic literature querying, because to be systematic you cannot be biased by the findings. I’m not saying this is bad, but it is a different tool for different jobs.

Here comes the catch, though.

There is always another player in the game of academia, the one that everyone hates but that somehow gets to make the rules. The publishers. A lot of journals are starting to request that literature reviews are systematic. Sometimes narrative reviews are accepted, but in different tracks, or only when authored by older, more expert professors (whereas literature reviews in my opinion are an excellent way to start a PhD, forcing the candidate to dive deep into the literature). The idea behind this is that science should be, well, scientific. Even meta-science, like a review. And being scientific requires systematicity, at least according to some. Personally I believe that different problems require different tools. It may be true that a narrative review may fail to be systematic, but on the other hand a systematic review fails to be as broad as one wishes.

Maybe the way forward would be to start calling the current systematic reviews as I suggest, systematic literature queries, and to find new ways to perform actual literature reviews in a systematic way. Or maybe, it would just be ok to keep on doing both, without discrediting either, and just knowing that they address different problems. Hopefully publishers will realize this as well.

Ad maiora!
- Filippo Carnovalini