Current Research

Meta-Analysis of Field Studies on Beauty and Professional Success

Irsova Zuzana, Havranek Tomas, Borntikova Kseniya, Bartoš František , Martina Luskova

R&R at Nature Human Behaviour

Abstract:

Common wisdom suggests that beauty helps in the labor market. We show that two factors combine to explain away most of the mean beauty premium reported in the literature. First, correcting for publication bias reduces the premium by at least a third. Second, controlling for cognitive ability renders the premium small (mean = 1.1%; 95% CrI = -0.8%, 3.0%) for all occupations except sex workers, where appearance is a direct input. The beauty premium is similar for earnings and productivity, a fact inconsistent with discrimination based on employer tastes for beauty. We find little evidence of attenuation bias that could offset publication and omitted-variable biases. To obtain these results we collect 1,159 estimates of the beauty premium in 67 studies and codify 35 aspects that reflect estimation context. We employ recently developed techniques to account for publication bias and model uncertainty.

PAP is available here

Publication Bias and p-Hacking in the Effect of COVID-19 on Learning

Luskova Martina, Buliskeria Nino, Eliminejad Ali, Havranek Tomas, Irsova Zuzana, Jurajda Stepan, Kapicka Marek

Abstract:

We revisit a central estimate in the economics of education: the human-capital loss associated with COVID-19 school closures. Estimates of pandemic learning loss may be affected by publication bias, p-hacking, and the mechanical correlation between standardized effect sizes and their standard errors. We conduct a comprehensive multi-method assessment of bias by applying a wide range of correction techniques — including PET-PEESE, three-parameter selection models (3PSM), Robust Bayesian Meta-Analysis (RoBMA), Meta-Analysis Instrumental Variable Estimation (MAIVE), Right-Truncated Meta-Analysis (RTMA), and multi-bias sensitivity analysis. Our preferred specifications, RoBMA and MAIVE, rely on different assumptions yet converge on an effect size of approximately -0.12 SD, equivalent to a learning loss of about 30% of a school year. Although some methods reveal signs of publication bias and selective reporting, these findings do not explain away the central finding: the COVID-19 learning deficit is economically meaningful and statistically robust.

PDF, code, and data are available here

Effect of Exercise on Cognition, Memory, and Executive Function: A Study-Level Meta-Meta-Analysis Across Populations and Exercise Categories

Bartos Frantisek, Luskova Martina, Bortnikova Kseniya, Hozova Karolina, Kantova Klara, Irsova Zuzana, Havranek Tomas

PsyArXiv pre-print

Abstract:

Physical exercise is widely believed to enhance cognition, yet evidence from meta-analysesremains mixed. Here we compile a study-level dataset of 2,239 effect-size estimates from215 meta-analyses of randomized controlled trials examining the effect of exercise ongeneral cognition, memory, and executive functions. We find strong evidence of selectivereporting and large between-study heterogeneity. Analyses adjusted for publication biasreveal average effects much smaller than commonly reported (general cognition:standardized mean difference, SMD, = 0.227, 95% credible interval 0.116 to 0.330; memory:SMD = 0.027, 95% credible interval 0.000 to 0.227; executive functions: SMD = 0.012, 95%credible interval 0.000 to 0.147), along with wide prediction intervals spanning bothnegative and positive effects. Subgroup analyses identify specific population-interventioncombinations with more consistent benefits. Overall, broad claims of generalized cognitiveenhancement resulting from physical exercise appear premature; the evidence supportstargeted, population- and intervention-specific recommendations.

PDF, code, and data are available here

Do decisions about outliers and influential effects matter? Evidence from 358 behavioral science meta-analyses

Havranek Tomas, Irsova Zuzana, Luskova Martina, TD Stanley

arXiv pre-print

Abstract:

Meta-analysts routinely face estimates that look too large or extreme. Yet, how to handle them is left to the reviewer's judgment. The methods for detecting such estimates are well known. What is missing is an informed assessment of how much alternative handling choices might change a meta-analysis' conclusions. We fill this gap by analyzing the effects of four pre-registered handling treatments across 358 behavioral science meta-analyses with at least ten estimates. Each outlier handling treatment is estimated by two estimators (random effects and unrestricted weighted least squares), and compared to the 'do-nothing' baseline on three outcomes: the pooled effect, statistical significance, and whether the effect reaches the smallest effect size of interest (|d| >= 0.20). Our entire analysis and comparison pipelines were pre-registered. Alternative outlier handling treatments have little effect on the meta-analysis mean as the median absolute change in Cohen's d is at most 0.047 and often much less. Yet, at least one of these four treatments in combination with one of these estimators reverses the statistical significance of 11.5% of meta-analyses and the smallest-effect-of-interest assessment in 15.9%. Winsorizing has the least effect and DFBETAS the most. Categorical changes are found almost entirely among results already close to the decision boundary; strongly significant results essentially never change. These findings give applied meta-analysts, methods specialists, and reviewers a reference point for how much this under-reported choice matters and provide yet another reason for meta-analysts to publicly pre-specify their methods and handling treatments.

PDF available here

A Comment on “A Systematic Review and Meta-analysis of the Evidence on Learning During the COVID-19 Pandemic”

Buliskeria Nino, Eliminejad Ali, Havranek Tomas, Irsova Zuzana, Jurajda Stepan, Kapicka Marek, Luskova Martina

IES working paper

Abstract:

Betthäuser et al. (2023) examine the effects of the COVID-19 pandemic on the learning progress of school-aged children. They collect 291 estimates from 42 studies. Their meta-analysis-corrected estimate implies a substantial decline in students’ learning (Cohen’s d = −0.14, 95% confidence interval −0.17 to −0.10). First, we successfully reproduce the main results and the majority of supporting figures. Second, we provide additional analysis addressing publication bias by implementing correction techniques: PET-PEESE (funnelbased), 3PSM (selection model), and RoBMA (model averaging). Additionally, we implement novel approaches that account for the strength of biased selection favoring affirmative results in the sample of analyzed studies. Third, we use techniques that assume the presence of p-hacking (MAIVE, RTMA). Using these methods, the corrected effect ranges from −0.25 to −0.11 with high statistical significance. While our analysis does reveal some evidence of publication bias and p-hacking, these phenomena do not appear to systematically distort the overall findings of the original study.

PDF is available here

Do Two Wrongs Make a Right? Publication and Attenuation Biases in Economics

Ioannidis John, Buliskeria Nino, Doucouliagos Chris, Elminejad Ali, Havranek Tomas, Irsova Zuzana, Luskova Martina, Stanley T. D.

PAP is available here

Bias-Correction Methods in Meta-Analysis: Prevalence and Effect on Estimates

Ioannidis John, Luskova Martina, Pardal Joanna

Disentangling p-Hacking and Publication Bias

Buliskeria Nino, Eliminejad Ali, Havranek Tomas, Irsova Zuzana, Luskova Martina