Skip to content
Accelerated ScienceJournals

Standards

Methods and Statistics

Methods are assessed on whether they adequately support the paper's claims.

What Is Assessed

For empirical research, the first priority is to establish a credible identification strategy. The validity of any result depends on whether the methods and assumptions justify interpreting the estimate as a measure of the effect or quantity the paper claims to identify. A reviewer examines the research design before anything else: what variation is being used, what would have to be true for the estimate to be causal and whether that assumption is plausible and defended rather than asserted.

  • Identification: what the design recovers, the assumption it rests on, and the evidence offered for that assumption.
  • Estimation and inference: whether the estimator matches the design, and whether uncertainty is reported honestly, including how standard errors are computed and clustered.
  • Robustness: whether the result survives the specifications a careful reader would ask for, and whether the reported analysis is the analysis described.
  • Data: provenance, sample construction and the exclusions applied.
  • Reproducibility: whether the result can be reproduced from the materials supplied.
  • Limitations: whether they are stated where they bear on the conclusion.

The Scope of Automated Verification Is Deliberately Limited

An automated statistical check is in development. Once in use, it will test internal consistency: whether the reported numbers, tables and magnitudes agree with the stated formulas and values. It is intended to catch transcription and arithmetic errors. It cannot judge identification, specification search, or whether an instrument is exogenous, and it will not be offered as if it could. Those are matters for the review panel.

What Is Not a Reason to Reject

A paper will not be rejected solely because its mathematics is simple or complex, its writing lacks elegance or its topic does not interest the reviewer. For a theory paper, strong assumptions are not themselves a fault: a model is entitled to its premises, provided they are stated and their consequences are drawn honestly. For an empirical paper, the position differs because an identifying assumption that is implausible is a defect in the result itself, and saying so is a valid ground for rejection.