Why reporting standards matter

Section 3.9 argues that transparent reporting and pre-analysis planning are not bureaucracy; they are credibility tools. They reduce researcher degrees of freedom, make design choices auditable, and let readers judge whether conclusions are robust or the result of specification search.

Design registries and timelines

A design registry is a timestamped record created before outcomes are analyzed. It should include:

  • The research question and estimand (e.g., ATT, $\tau(g,t)$, $\theta_k$, or $\mu(d)$).
  • The assignment mechanism and treatment window.
  • The primary estimator and inference plan.
  • Data sources, cleaning rules, and transformations.

Timelines complement registries by documenting when treatment was assigned, when outcomes were measured, and when the analysis plan was finalized. Version-controlled scripts (including failed specifications) help distinguish pre-specified choices from exploratory analysis.

When pre-registration is infeasible: Use timestamped scripts, holdout samples, and clear labeling of exploratory vs pre-specified analyses. These are imperfect substitutes but still constrain flexibility.

Assignment matrices and exposure maps

The assignment matrix $D=(D_{it})$ makes treatment timing and intensity explicit. It should be shared in supplementary materials or in an auditable repository. This is essential for:

  • Verifying the assignment protocol.
  • Replicating cohort definitions.
  • Diagnosing not-yet-treated controls in rollouts.

When interference is plausible, also publish exposure maps that define $h_i(D_{-i,t})$. Exposure maps clarify assumptions and guide sensitivity analysis.

Outcome definitions aligned to estimands

Outcome definitions must match the estimand. If the estimand is incremental sales, the outcome should measure incremental sales, not total revenue. If the estimand is long-run value, short-run revenue is misaligned.

Always specify:

  • The measurement window.
  • Any adjustments (seasonal, inflation, scaling).
  • The avoidance of post-treatment controls.

Primary vs secondary outcomes

Pre-specify a primary outcome (the one you power and test formally) and mark secondary outcomes as exploratory. This prevents ex post promotion of a secondary metric simply because it showed a large effect.

This also defines the family for multiplicity adjustment.

Inference and multiplicity plans

Pre-analysis plans should specify:

  • Standard error type (unit clustering, time clustering, two-way clustering).
  • Use of randomization inference or wild cluster bootstrap.
  • How multiple comparisons will be handled.

Multiplicity guidance:

  • Bonferroni for a small set of pre-specified hypotheses.
  • Holm-Bonferroni for a less conservative family-wise approach.
  • FDR control for exploratory families with many tests.

For event-study profiles $\{\theta_k\}$ or large sets of cohort-time effects $\{\tau(g,t)\}$, prefer joint statements (confidence bands or omnibus tests) over dozens of pointwise claims.

Align power with the inference plan

Power calculations must match the planned inference. If the analysis uses two-way clustering or randomization inference, power should be computed under those assumptions. If multiplicity adjustments will be used, power should incorporate them.

Handling deviations from plan

If you deviate from the pre-analysis plan, document the deviation and report both the planned and revised analyses. Do not overwrite the original plan. Transparency is the point.

Practical reporting checklist

  • Pre-registered design registry with estimands and estimator.
  • Timestamped assignment matrix and exposure map.
  • Pre-specified outcomes, windows, and controls.
  • Primary vs secondary outcomes clearly labeled.
  • Inference and multiplicity plan aligned with power calculations.
  • Deviations logged and justified, not hidden.

Takeaway

Pre-analysis plans and reporting standards do not make a design perfect, but they make it auditable. In marketing panels where identification hinges on assumptions, transparency about design and analysis choices is itself part of the evidence.

References

  • Shaw, C. (2025). Causal Inference in Marketing: Panel Data and Machine Learning Methods (Community Review Edition), Section 3.9.
  • Simmons, J. P., Nelson, L. D., and Simonsohn, U. (2011). False-positive psychology.
  • Benjamini, Y., and Hochberg, Y. (1995). Controlling the false discovery rate.
  • Holm, S. (1979). A simple sequentially rejective multiple test procedure.