Reviewer Guidelines

The quality of published articles largely depends on the quality of the reviews that authors receive. To help make reviewer reports more consistent, we have prepared a short list of guiding questions for reviewers to consider. This list is not exhaustive, nor is it applicable to all submissions, as manuscripts may have different aims and methodological approaches.

General Reviewer Guidelines

Rationale

  • Is there a rationale for the manuscript provided (the fact that something has not been done is insufficient)?
  • What is the purpose of the measurement instrument (i.e., what it intends to measure and why)?

Construct definition

  • Is the construct defined in an unambiguous way?
  • Is the relevant theoretical literature cited?

Target population and samples

  • Is the target population defined?
  • Is the sample well described (type of sample, dates of data collection, sample composition)?
  • Does the sample allow for generalization to the target population?

Validity

  • Does the instrument (i.e., the items) capture all relevant attributes of the construct?
  • Does the empirical structure of the instrument match the theoretical structure of the construct (e.g., with respect to the number of dimensions)?
  • Is there sufficient evidence for convergent and divergent validity (i.e., does the instrument correlate with external variables in an expected way, and is the set of correlates sufficient to establish divergent and convergent validity)?

Reliability

  • Is there evidence for reliability reported?

Statistical procedures

  • Is the methodological detail sufficient to replicate the analysis?
  • Are the methods used in the article congruent with the current state-of-the-art?
  • Are the proportion of missing data and the approach for handling them reported?

 

Specifically for new instruments:

  • What sources of information were used during item generation (e.g., existing scales, literature research, focus groups, expert opinion)?
  • Based on which selection criteria were items included in (or excluded from) the final version of the instrument (e.g., item characteristics and/or theoretical considerations)?
  • How were items generated (e.g., four experts wrote them)?
  • Were any pretests conducted (e.g., cognitive pretests, webprobing) to ensure the comprehensibility of the items?