Methods 1 , 66—80 Egger, M. Bias in meta-analysis detected by a simple, graphical test. Duval, S. Trim and fill: a simple funnel-plot-based method of testing and adjusting for publication bias in meta-analysis. Biometrics 56 , — Leimu, R. Cumulative meta-analysis: a new tool for detection of temporal trends and publication bias in ecology. B , — Higgins, J. This large collaborative work provides definitive guidance for the production of systematic reviews in medicine and is of broad interest for methods development outside the medical field.
Lau, J. Lortie, C. How to critically read ecological meta-analyses. Methods 6 , — Murad, M. Synthesizing evidence: shifting the focus from individual studies to the body of evidence. Rasmussen, S. Maternal obesity and risk of neural tube defects: a meta-analysis. Multisystemic therapy for social, emotional, and behavioral problems in youth aged 10— Cochrane Database Syst. Schmidt, F. What do data really mean? Research findings, meta-analysis, and cumulative knowledge in psychology. Button, K.
Power failure: why small sample size undermines the reliability of neuroscience. Parker, T.
- Meta-analyses were supposed to end scientific debates. Often, they only cause more controversy.
- Bibliographic Information.
- Viewing systematic reviews and meta-analysis in social research through different lenses.
- MetaArXiv Preprints | Search;
- The Messiah Eyes: An Experience.
Transparency in ecology and evolution: real problems, real solutions. Trends Ecol. Stewart, G. Meta-analysis in applied ecology. Sutherland, W. The need for evidence-based conservation. Lowry, E. Biological invasions: a field synopsis, systematic review, and database of the literature. Parmesan, C. A globally coherent fingerprint of climate change impacts across natural systems. Nature , 37—42 Balvanera, P. Quantifying the evidence for biodiversity effects on ecosystem functioning and services. Cardinale, B.
Advances in Meta-Analysis
Effects of biodiversity on the functioning of trophic groups and ecosystems. Nature , — Rey Benayas, J. Enhancement of biodiversity and ecosystem services by ecological restoration: a meta-analysis. Science , — How general are positive relationships between plant population size, fitness and genetic variation? On the generality of the latitudinal diversity gradient.
Gurevitch, J. Rustad, L. A meta-analysis of the response of soil respiration, net nitrogen mineralization, and aboveground plant growth to experimental ecosystem warming. Oecologia , — Adams, D. Phylogenetic meta-analysis. Evolution 62 , — Hadfield, J. General quantitative genetic methods for comparative biology: phylogenies, taxonomies and multi-trait models for continuous and categorical characters.
Lajeunesse, M. Meta-analysis and the comparative phylogenetic method. Rosenberg, M. Wallace, B. OpenMEE: intuitive, open-source software for meta-analysis in ecology and evolutionary biology. Methods Ecol. The interaction between competition and predation: a meta-analysis of field experiments. Resampling tests for meta-analysis of ecological data. Ecology 78 , — Statistical issues in ecological meta-analyses. Ecology 80 , — Schmid, C. Eysenck, H. Exercise in mega-silliness.
Simberloff, D. Cadotte, M. Gauging the impact of meta-analysis on ecology. The case of the misleading funnel plot. Vetter, D. Meta-analysis: a need for well-defined usage in ecology and conservation biology. Ecosphere 4 , 1—24 Mengersen, K. Patsopoulos, N. Relative citation impact of various study designs in the health sciences.
- Log in to Wiley Online Library;
- Boxed Set-Coming Full Circle & Randalls Forever Love & Love Is a Gamble!
- Alone in a Crowd (Cattle Valley Book 27).
Kueffer, C. Fame, glory and neglect in meta-analyses.
Titles in this series
Cohnstaedt, L. Review Articles: The black-market of scientific currency. Longo, D. Data sharing. Gauch, H. Scientific Method in Practice Cambridge Univ. Science Staff. Dealing with data: introduction. Challenges and opportunities. Nosek, B. Promoting an open research culture. Stewart, L. Saldanha, I. Evaluating Data Abstraction Assistant, a novel software application for data abstraction during systematic reviews: protocol for a randomized controlled trial. Tipton, E. Small-sample adjustments for tests of moderators and model fit using robust variance estimation in meta-regression.
The potential for meta-analysis to support decision analysis in ecology. Ashby, D. Bayesian statistics in medicine: a 25 year review. Senior, A. Heterogeneity in ecological and evolutionary meta-analyses: its magnitude and implications. Ecology 97 , — McAuley, L. Does the inclusion of grey literature influence estimates of intervention effectiveness reported in meta-analyses?
Lancet , — Press, This book provides the first comprehensive guide to undertaking meta-analyses in ecology and evolution and is also relevant to other fields where heterogeneity is expected, incorporating explicit consideration of the different approaches used in different domains. Lumley, T. Network meta-analysis for indirect treatment comparisons. Zarin, W. Characteristics and knowledge synthesis approach for network meta-analyses: a scoping review. BMC Med. Elliott, J. Living systematic reviews: an emerging opportunity to narrow the evidence-practice gap.
What other items do customers buy after viewing this item?
Vandvik, P. Living cumulative network meta-analysis to reduce waste in research: a paradigmatic shift for systematic reviews? Jarvinen, A. A meta-analytic study of the effects of female age on laying date and clutch size in the Great Tit Parus major and the Pied Flycatcher Ficedula hypoleuca. Ibis , 62—67 Arnqvist, G.
Meta-analysis: synthesizing research findings in ecology and evolution. The meta-analysis of response ratios in experimental ecology. Meta-analysis in ecology. Res 32 , — Bioinformatics 27 , — Pearson, K. Report on certain enteric fever inoculation statistics. Fisher, R. Yates, F. The analysis of groups of experiments. Cochran, W. The combination of estimates from different experiments. Biometrics 10 , — Smith, M. Meta-analysis of psychotherapy outcome studies. Glass, G. Meta-analysis at middle age: a personal history. Cooper, H.
Statistics and Its Interface
This book is an important compilation that builds on the ground-breaking first edition to set the standard for best practice in meta-analysis, primarily in the social sciences but with applications to medicine and other fields. Rosenthal, R. Meta-analytic Procedures for Social Research Sage, The average effect size across studies can be used to summarize the strength and direction of the entire body of findings.
If the reviewer is interested in examining study findings as a function of the substantive and methodological characteristics of the studies, the effect size from each individual study can serve as the dependent variable in such an analysis. When correlational rather than experimental studies are being integrated, the above direct measure of effect size is obviously not amenable to meta-analysis.
The results of individual studies will usually be represented by correlation coefficients. Values of r or r2 or some transformation of them will enter the meta-analysis as dependent variables. When a meta-analysis involves independent variables representing study characteristics, the measurement of the independent variables obviously becomes an issue. Unfortunately, there are no straightforward a priori guidelines for determining or measuring the independent variables of a meta-analysis. Three-general points can be made, however. First, determination and measurement of variables for meta-analytic purposes requires that the studies be investigated before the variables are selected.
In other words, the determination of study characteristics to be included can only occur after the studies have been screened for commonalities as well as differences. With the knowledge obtained the reviewer can then identify independent variables for study and develop a coding scheme for them.
Preprint Archive Search
This is why an active researcher in the area may well be a more efficient and appropriate meta-analyst. Once the independent variables have been determined the reliability and validity of their representation must be considered. Thus, the second and third key points deal with the psychometric aspects of a meta-analysis. Reliability issues in a meta-analysis can be thought of in terms of intercoder reliability.
Steps must be taken to ensure unambiguous coding instructions in order to reduce coding errors. Finally, the validity of meta-analytic measures is perhaps their most elusive aspect of measurement. There are no technical procedures specifically designed for assessing the validity of meta-analytic measurement, although some conventional procedures may be capable of being extended to this type or research.
Insights into validity issues may be obtained from a thorough reading of existing meta-analyses before beginning data collection. Meta-analytic sampling is concerned with the selection or studies to be included in the quantitative review. Rules of sampling that apply to survey research are relevant here. At a minimum, a representative sample of studies is desired. Yet the population of interest must be determined. Thus, the meta-analyst should probably think in terms of attempting a census of studies concerning the research question. It is in the development or the sampling frame where controversy surrounding meta-analytic sampling arises.
The issue lies in whether all studies on a relationship should be eligible or only those meeting certain criteria e. Glass, McGaw, and Smith offer the most encompassing resolution of this issue. They argue that all studies be eligible for inclusion and that differences between studies regarding age, quality, etc. This approach results in a more complete meta-analysis but does magnify the search for and coding of studies. The essence or meta-analysis is data analysis. It is the reason that studies were brought together in the first place--to statistically summarize their findings as a body of data in and of itself and to extract meaning from it.
The scope of statistical procedures available to the primary researcher is available to the meta-analyst. Depending on the research question, univariate, bivariate, or multivariate techniques can be used in a meta-analysis. Moreover, the reviewer's perspective on the data can be exploratory or confirmatory, descriptive or inferential. Whatever the perspective, meta-analysis seeks data reduction for the purpose of interpretation.
The primary objective is to synthesize the literature on a topic and, hopefully, offer generalizations concerning the current status and future needs for research in the area. The intended outcome of a meta-analysis is a more objective, impartial basis for interpreting the findings of many studies than a narrative approach provides. Nonetheless, a meta-analysis and a meta-analyst are not without imperfections and limitations at this stage. Meta-analysis should not be accepted as providing generalizations beyond the body of research that was reviewed.
For example, if only laboratory studies were reviewed, a meta-analysis of their findings does not make these findings generalizable to other settings. Finally, the liberties the meta-analyst has in attaching meaning to statistical results is no greater than in primary research. As Cotton and Cook , p. Glass et al. Sudman and Bradburn performed a meta-analysis on several hundred studies of response effects in surveys.
A total of 46 independent variables were coded and divided into three groups: task variables, interviewer role, and respondent role. These variables were investigated to determine their ability to explain response effects in a variety of marketing and other social science studies. As a measure of relative effect size, Sudman and Bradburn computed an index of how much difference a particular variable e.
By employing this formal, quantitative approach, it was determined that in general, the variables derived from the nature and structure of the task are more important than respondent or interviewer characteristics. The authors also pointed out that unlike qualitative approaches, this quantitative approach gives a less biased indication of the importance of variables, forces more careful attention to variable definition, and most importantly, allows ranking or the importance of the independent variables. This study well illustrates the value of meta-analysis for methodological as well as substantive research issues.
Schwab, et al. One-hundred sixty observations were derived from 39 studies.
By combining studies quantitatively and investigating these methodological concerns, the authors supported "the nagging suspicion that expectancy theory over-intellectualizes the cognitive processes people go through when choosing alternative actions at least insofar as choosing a level of performance or effort is concerned " p. A marketing meta-analysis that focused on effect size was Clarke's review of research assessing the duration of advertising effects on sales. Clarke analyzed 69 studies, which included some for which the effects of advertising were not statistically significant.
After the elimination of eleven studies that yielded duration estimates longer than ten years which Clarke believed implied a nonsensical result , the quantitative meta-analysis yielded several important insights not available from a more traditional qualitative literature review e. First, the results indicated that the estimate of the duration of advertising effect was contingent upon the data interval. Shorter intervals weekly, monthly, or bimonthly indicated shorter estimates of the duration of advertising effects than longer data intervals quarterly, annually.
From econometrics and advertising theory, Clarke devised a test for interval bias; this test suggested that the annual data intervals were more apt to produce estimates that were biased upwards. Perhaps most important, Clarke was able to conclude that, contrary to past beliefs, advertising effects are likely to last for no more than three to nine months and not years. Clarke summarized by stating that, although he had to make some subjective decisions in order to produce comparable model specifications, "in isolation, none of the papers gives a satisfactory answer to the question of how long advertising affects sales.
By putting them together, as has been done here, one achieves greater confidence in the result" p. Yu and Cooper forthcoming conducted a meta-analysis of techniques used to increase response races to questionnaires. Conclusions were drawn by combining response races found in 93 journal articles.
Both the statistical significance of the accumulated results for a particular technique and the effect size phi-coefficient associated with a technique were computed. It was found that response races were increased by personal and telephone versus mail surveys, the use of prepaid or promised incentives, nonmonetary premiums and rewards and increasing amounts of monetary rewards. Other facilitators were preliminary notification, foot-in-the-door techniques, personalization and follow-up letters. The authors noted that the vast literature on the topic of response races makes "qualitative reviews extremely difficult to perform and their results necessarily imprecise in nature" and that quantitative reviewing helps increase the objectivity and reliability or review conclusions.
Two consumer research studies which can be considered under the rubric of meta-analysis were conducted by Farley, Lehmann and Ryan a, b. In Farley et al. The three criterion variables were the average beta weights for attitudinal and normative components and the average multiple correlation coefficient.
The five predictor variables were the form of the attitudinal variable, the form of the normative variable, whether the study was experimental or not, the researcher's dominant discipline affiliation and whether subjects were students or "real world" respondents. Of the five main effects in the analysis, only the discipline of the researcher was concluded to have a large effect.
In Farley, et al. Few elasticities were found to differ from the overall mean other than those associated with controllable exogenous variables such as price and distribution. Situational factors such as socio-demographics and study-specific factors had little impact on the elasticities.
A final example of used systematic approach to review is Hyde's meta-analysis of previous studies of whether males or females are superior in terms of several dimensions of cognitive ability. Previous qualitative literature reviews had concluded that differences in various abilities were "well-established. Hyde found 27 studies or verbal ability, 16 studies of quantitative ability, 10 studies of visual-spatial ability, and 20 studies of field articulation. For the studies which offered sufficient information to calculate the effect size estimates, the respective median X and d values were.
Hyde suggested that of the traditional qualitative literature reviews based simply or the number of studies which found statistically significant results may have misleadingly communicated the impression that the moderately consistent statistically significant sex differences were large when in tact they explained only from 1 to 42 or the variance and averaged less than. Hyde concluded that, "Of course, a small effect might still be an important one.
But at least the reader would have the option of deciding whether a statistically significant effect was large enough to merit further attention, either in teaching or in research" p. These examples illustrate the wide range of topics which can be addressed with meta-analysis as well as the wide range of statistical procedures which can be employed.
In addition, they illustrate how both substantive and methodological factors can be used to attempt to explain research findings. There are a number of problems in conducting meta-analyses. Among these difficulties are the quantification, interpretation, and generalization of various types of effect size measures.
For example, some such measures estimate the ratio or explained to total variance such as R2 or w2. In quantifying the percentage of explained variance, researchers should recognize that total variance is increased by measurement and treatment unreliability, heterogeneous subjects, and poorly controlled research procedures Sechrest and Yeaton a,b.
Experimental researchers can also influence the amount of explained variance by restricting or magnifying the manipulation of an independent variable. Independent variables which are qualitative or categorical present particular interpretation problems. Such variables often have no conceptually meaningful or practically important characteristics in common within or across studies; the number of "levels" of such variables is infinite and any estimates of the "size" of their effects are very difficult to interpret.
Finally, although estimates of percentage of explained variance may provide a common index for comparison, the above problems of the influence of individual characteristics of particular studies and manipulations within a study make it very difficult to meaningfully generalize effect sizes or to compare them across a set of different studies as in a meta-analysis. However effect sizes are estimated, these descriptive statistics are more generalizable if the levels of the independent variables are a random subset of all levels of interest Glass and Hakstian and orthogonal to other independent variables Green, Carroll and DeSarbo ; LaTour a.
Fortunately, other approaches and measures of effect size are available for quantitatively summarizing research. As previously noted, Rosenthal has discussed the advantages and limitations of nine relatively simple approaches to summarizing results. LaTour a, b recommends the use of a contrast estimate to quantify effect size since it eliminates many of the problems of explained variance estimates. However, these methods seem most appropriate for the common 2 x 2 research design and are difficult to use and interpret with more complex designs Glass and Hakstian Glass, McGaw and Smith , p.
In addition to the problem or meaningfully summarizing and comparing study results, a meta-analysis often encounters other formidable obstacles. One problem involves the search for a census of studies including the unpublished ones that likely have smaller effect sizes. For studies that are available, there is often insufficient information to be able to calculate effect sizes and study authors must be contacted.
Unfortunately, it is also often difficult to obtain sufficiently detailed descriptions of study methods and to code these study characteristics so their effects can be assessed in the meta-analysis. Small samples of studies and confounded study characteristics also make it difficult to disentangle main effects. See Farley, Lehmann and Ryan b for an example of how to deal with the potential negative degrees of freedom issue. Main effects across studies are much easier to detect than most complex interactions.
An opposite problem is that, if all surveyed studies use the same procedure, the effect of that method cannot be assessed e. One important outcome of a meta-analysis might be a specification of types or studies that would fill an existing void and allow an examination of the effects of variables that cannot currently be meaningfully evaluated. It should now be obvious that a meta-analysis, though quantitative, depends on many subjective researcher decisions and there is much opportunity for disagreement e. Perhaps because the publication of a meta-analysis carries an aura or finality, it seems very common for researchers to disagree about the many decisions involved in a meta-analysis and, hence, challenge the conclusions.
For example, Stanley and Benbow challenged Hyde's meta-analysis of gender differences in quantitative ability. By analyzing only males and females who achieved high scores on a standardized mathematical achievement test, Stanley and Benbow found that males were much more likely to score high than females. They conclude that, "It seems to us that much research into causes and remedies is sorely needed, rather than further efforts trying to minimize the magnitude of sex differences" p.
Weinberg and Weiss have disputed some of the analysis decisions in Clarke's meta-analysis of advertising carryover as well as the statistical validity of his conclusion about data interval bias. Weinberg and Weiss's criticisms include a failure of Clarke's analysis to allow for situational contingencies such as product class, the combination of brand loyal models with Koyck models when the former do not distinguish nonadvertising effects from advertising effects, model misspecification, and a publication bias in favor of statistically significant advertising carryover effects which, in turn, are related to the data aggregation level Weiss and Windall