How should software engineering secondary studies include grey material?

Context: Recent papers have proposed the use of grey literature (GL) and multivocal reviews. These papers have raised issues about the practices used for systematic reviews (SRs) in software engineering (SE) and suggested that there should be changes to the current SR guidelines.

Objective: To investigate whether current SR guidelines need to be changed to support GL and multivocal reviews.

Method: We discuss the definitions of GL and the importance of GL and of industry-based field studies in SE SRs. We identify properties of SRs that constrain the material used in SRs: a) the nature of primary studies b) the requirements of SRs to be auditable, traceable, and reproducible, and explain why these requirements restrict the use of blogs in SRs.

Results: SR guidelines have always considered GL as a possible source of primary studies and never supported excluding field studies that incorporate the practitioners’ viewpoint. However, GL which was meant to refer to documents that were not formally published, is now being extended to information from blogs/tweets/Q&A posts. Thus, it might seem that SRs do not make full use of GL because they do not include such information. However, the unit of analysis in SR is the primary study. Thus, it is not the source but the type of information that is important. Any report describing a rigorous empirical evaluation is a candidate primary study. Whether it is actually included in a SR depends on the SR eligibility criteria. However, any study that cannot be guaranteed to be publicly available in the long term should not be used as a primary study in an SR. This does not prevent such information being aggregated in surveys of social media and used in the context of evidence-based software engineering (EBSE).

Conclusions: Current guidelines for SRs do not require extensions, but their scope needs to be better defined. SE researchers require guidelines for analysing social media posts (e.g., blogs, tweets, vlogs), but these should be based on qualitative primary (not secondary) study guidelines. SE researchers can use mixed-methods SRs and/or the fourth step of EBSE to incorporate findings from social media surveys with those from SRs and to develop industry-relevant recommendations.