Standards
Methods and statistics
Methods are assessed on whether they support the claims made from them.
What is assessed
For an empirical paper, identification comes first. A result is only as good as the argument that the estimate recovers the quantity the paper claims it recovers. A reviewer examines the research design before anything else: what variation is being used, what would have to be true for the estimate to be causal, and whether that assumption is plausible and defended rather than asserted.
- Identification: what the design recovers, the assumption it rests on, and the evidence offered for that assumption.
- Estimation and inference: whether the estimator matches the design, and whether uncertainty is reported honestly, including how standard errors are computed and clustered.
- Robustness: whether the result survives the specifications a careful reader would ask for, and whether the reported analysis is the analysis described.
- Data: provenance, sample construction, and the exclusions applied.
- Reproducibility: whether the result can be reproduced from the materials supplied.
- Limitations: whether they are stated where they bear on the conclusion.
The machine checks are narrow, deliberately
The automated statistical check tests internal consistency: whether the reported numbers, tables and magnitudes agree with the stated formulas and values. It catches transcription and arithmetic errors. It cannot judge identification, specification search, or whether an instrument is exogenous, and it is not offered as if it could. Those are matters for the subject reviewer.
What is not a reason to reject
That the mathematics is simple or difficult, that the paper is inelegant, or that the topic does not interest the reviewer. For a theory paper, strong assumptions are not themselves a fault: a model is entitled to its premises, provided they are stated and their consequences are drawn honestly. For an empirical paper the position differs, because an identifying assumption that is implausible is a defect in the result itself, and saying so is a valid ground for rejection.