Rethinking the Role of Teams and Training in Geopolitical Forecasting: The Effect of Uncontrolled Method Variance on Statistical Conclusions.

Using data from a geopolitical forecasting tournament, Mellers et al. (2014) [Psychological strategies for winning a geopolitical forecasting tournament. , 1106-1115] concluded that forecasting ability was improved by allowing participants to work in teams and providing them with probability training.

A memory-theoretic account of citation propagation.

Despite the common assumption that citations are indicative of an article's scientific merit, increasing evidence indicates that citation counts are largely driven by variables unrelated to quality. In this article, we treat people's decisions of what to cite as an instance of memory retrieval and show that observed citation patterns are well accounted for by a model of memory. The proposed exposure model anticipates that small alterations in factors that affect people's ability to retrieve to-be-cited articles from memory early in their life cycle are magnified over time and can lead to the emergence of highly cited papers.

Citation counts and journal impact factors do not capture some indicators of research quality in the behavioural and brain sciences.

Citation data and journal impact factors are important components of faculty dossiers and figure prominently in both promotion decisions and assessments of a researcher's broader societal impact. Although these metrics play a large role in high-stakes decisions, the evidence is mixed about whether they are strongly correlated with indicators of research quality. We use data from a large-scale dataset comprising 45 144 journal articles with 667 208 statistical tests and data from 190 replication attempts to assess whether citation counts and impact factors predict three indicators of research quality: (i) the accuracy of statistical reporting, (ii) the evidential value of the reported data and (iii) the replicability of a given experimental result.

Making Research Evaluation More Transparent: Aligning Research Philosophy, Institutional Values, and Reporting.

There is a growing interest in changing the culture of psychology to improve the quality of our science. At the root of this interest is concern over the reproducibility of key findings. A variety of large-scale replication attempts have revealed that several previously published effects cannot be reproduced, whereas other analyses indicate that the published literature is rife with underpowered studies and publication bias.

Do metacognitive judgments alter memory performance beyond the benefits of retrieval practice? A comment on and replication attempt of Dougherty, Scheck, Nelson, and Narens (2005).

A central question in the metacognitive literature concerns whether the act of making a metacognitive judgment alters one's memory for the information about which the judgment was made. Dougherty, Scheck, Nelson, and Narens (2005, Memory & Cognition, 33(6), 1096-1115) attempted to address this question by having participants make either retrospective confidence judgments (RCJs; i.e.

Crowdsourcing prior information to improve study design and data analysis.

Though Bayesian methods are being used more frequently, many still struggle with the best method for setting priors with novel measures or task environments. We propose a method for setting priors by eliciting continuous probability distributions from naive participants. This allows us to include any relevant information participants have for a given effect.

Making Retrospective Confidence Judgments Improves Learners' Ability to Decide What Not to Study.

Predictions about future retrieval success, known as judgments of learning (JOLs), are often viewed as important for effective control over learning. However, much less is known about how retrospective confidence judgments (RCJs), evaluations of past retrieval success, may affect control over learning. We compared participants' ability to identify items that would benefit from additional study after making either a JOL or an RCJ.

Order-constrained linear optimization.

Despite the fact that data and theories in the social, behavioural, and health sciences are often represented on an ordinal scale, there has been relatively little emphasis on modelling ordinal properties. The most common analytic framework used in psychological science is the general linear model, whose variants include ANOVA, MANOVA, and ordinary linear regression. While these methods are designed to provide the best fit to the metric properties of the data, they are not designed to maximally model ordinal properties.

Does constraining memory maintenance reduce visual search efficiency?

We examine whether constraining memory retrieval processes affects performance in a cued recall visual search task. In the visual search task, participants are first presented with a memory prompt followed by a search array. The memory prompt provides diagnostic information regarding a critical aspect of the target (its colour).

Reevaluating the effectiveness of n-back training on transfer through the Bayesian lens: Support for the null.

A recent meta-analysis by Au et al. Psychonomic Bulletin & Review, 22, 366-377, (2015) reviewed the n-back training paradigm for working memory (WM) and evaluated whether (when aggregating across existing studies) there was evidence that gains obtained for training tasks transferred to gains in fluid intelligence (Gf). Their results revealed an overall effect size of g = 0.

Using a model of hypothesis generation to predict eye movements in a visual search task.

We used a model of hypothesis generation (called HyGene; Thomas, Dougherty, Sprenger, & Harbison, 2008) to make predictions regarding the deployment of attention (as assessed via eye movements) afforded by the cued recall of target characteristics before the onset of a search array. On each trial, while being eyetracked, participants were first presented with a memory prompt that was diagnostic regarding the target's color in a subsequently presented search array. We assume that the memory prompts led to the generation of hypotheses (i.

Measuring working memory is all fun and games: a four-dimensional spatial game predicts cognitive task performance.

We developed a novel four-dimensional spatial task called Shapebuilder and used it to predict performance on a wide variety of cognitive tasks. In six experiments, we illustrate that Shapebuilder: (1) Loads on a common factor with complex working memory (WM) span tasks and that it predicts performance on quantitative reasoning tasks and Ravens Progressive Matrices (Experiment 1), (2) Correlates well with traditional complex WM span tasks (Experiment 2), predicts performance on the conditional go/no go task (Experiment 3) and N-back (Experiment 4), and showed weak or nonsignificant correlations with the Attention Networks Task (Experiment 5), and task switching (Experiment 6). Shapebuilder shows that it exhibits minimal skew and kurtosis, and shows good reliability.

Reply to Rouder (2014): good frequentist properties raise confidence.

Established psychological results have been called into question by demonstrations that statistical significance is easy to achieve, even in the absence of an effect. One often-warned-against practice, choosing when to stop the experiment on the basis of the results, is guaranteed to produce significant results. In response to these demonstrations, Bayes factors have been proposed as an antidote to this practice, because they are invariant with respect to how an experiment was stopped.

What counts as evidence for working memory training? Problems with correlated gains and dichotomization.

The question of whether computerized cognitive training leads to generalized improvements of intellectual abilities has been a popular, yet contentious, topic within both the psychological and neurocognitive literatures. Evidence for the effective transfer of cognitive training to nontrained measures of cognitive abilities is mixed, with some studies showing apparent successful transfer, while others have failed to obtain this effect. At the same time, several authors have made claims about both successful and unsuccessful transfer effects on the basis of a form of responder analysis, an analysis technique that shows that those who gain the most on training show the greatest gains on transfer tasks.

When decision heuristics and science collide.

The ongoing discussion among scientists about null-hypothesis significance testing and Bayesian data analysis has led to speculation about the practices and consequences of "researcher degrees of freedom." This article advances this debate by asking the broader questions that we, as scientists, should be asking: How do scientists make decisions in the course of doing research, and what is the impact of these decisions on scientific conclusions? We asked practicing scientists to collect data in a simulated research environment, and our findings show that some scientists use data collection heuristics that deviate from prescribed methodology. Monte Carlo simulations show that data collection heuristics based on p values lead to biases in estimated effect sizes and Bayes factors and to increases in both false-positive and false-negative rates, depending on the specific heuristic.

Retrieval dynamics in self-terminated memory search.

Most free-recall experiments employ a paradigm in which participants are given a preset amount of time to retrieve items from a list. While much has been learned using this paradigm, it ignores an important component of many real-world retrieval tasks: the decision to terminate memory search. The present study examines the temporal characteristics underlying memory search by comparing within subjects a standard retrieval paradigm with a finite, preset amount of time (closed interval) to a design that allows participants to terminate memory search on their own (open interval).

Robust decision making in a nonlinear world.

The authors propose a general modeling framework called the general monotone model (GeMM), which allows one to model psychological phenomena that manifest as nonlinear relations in behavior data without the need for making (overly) precise assumptions about functional form. Using both simulated and real data, the authors illustrate that GeMM performs as well as or better than standard statistical approaches (including ordinary least squares, robust, and Bayesian regression) in terms of power and predictive accuracy when the functional relations are strictly linear but outperforms these approaches under conditions in which the functional relations are monotone but nonlinear. Finally, the authors recast their framework within the context of contemporary models of behavioral decision making, including the lens model and the take-the-best heuristic, and use GeMM to highlight several important issues within the judgment and decision-making literature.

Generating and evaluating options for decision making: the impact of sequentially presented evidence.

We examined how decision makers generate and evaluate hypotheses when data are presented sequentially. In the first 2 experiments, participants learned the relationship between data and possible causes of the data in a virtual environment. Data were then presented iteratively, and participants either generated hypotheses they thought caused the data or rated the probability of possible causes of the data.

Implications of cognitive load for hypothesis generation and probability judgment.

We tested the predictions of HyGene (Thomas et al., 2008) that both divided attention at encoding and judgment should affect the degree to which participants' probability judgments violate the principle of additivity. In two experiments, we showed that divided attention during judgment leads to an increase in subadditivity, suggesting that the comparison process for probability judgments is capacity limited.

On the lawfulness of the decision to terminate memory search.

Nearly every memory retrieval episode ends with a decision to terminate memory search. Yet, no research has investigated whether these search termination decisions are systematic, let alone whether they are made consistent with a particular rule. In the present paper, we used a modified free-recall paradigm to examine the decision to terminate search.

Psychological plausibility of the theory of probabilistic mental models and the fast and frugal heuristics.

The theory of probabilistic mental models (PMM; G. Gigerenzer, U. Hoffrage, & H.

Diagnostic hypothesis generation and human judgment.

Diagnostic hypothesis-generation processes are ubiquitous in human reasoning. For example, clinicians generate disease hypotheses to explain symptoms and help guide treatment, auditors generate hypotheses for identifying sources of accounting errors, and laypeople generate hypotheses to explain patterns of information (i.e.

Motivated to retrieve: how often are you willing to go back to the well when the well is dry?

Despite the necessity of the decision to terminate memory search in many real-world memory tasks, little experimental work has investigated the underlying processes. In this study, the authors investigated termination decisions in free recall by providing participants an open-ended retrieval interval and requiring them to press a stop button when they had finished retrieving. Three variables important to assessing one's willingness to search memory were examined: (a) the time spent searching memory after the last successful retrieval before choosing to quit (the exit latency); (b) task difficulty; and (c) individual differences in motivation, as measured by Webster and Kruglanski's (1994) Need for Closure Scale.

The effect of divided attention on global judgment of learning accuracy.

This study examined the effect of divided attention (DA) on global judgment of learning (JOL) accuracy in a multitrial list learning paradigm. A word monitoring task was used to divide attention. Participants were assigned to an attention condition (DA at encoding, DA at judgment, DA at retrieval, or focused attention) and completed 4 learning trials, each comprising a study, judgment, and recall phase.

Word concreteness and encoding effects on context-dependent discrimination.

In three experiments, we investigated the effects of word concreteness and encoding instructions on context-dependent discrimination in verbal contexts, using Murnane, Phelps, and Malmberg's (1999) ICE (item, context, ensemble) theory as the framework Word concreteness was manipulated within participants, and encoding was manipulated between participants. It was hypothesized that the magnitude of context-dependent discrimination would be affected by both concreteness and encoding instructions. Imagery instructions resulted in context-dependent discrimination for both concrete and abstract word pairs across all the experiments.