Abstract
When evaluating experimental evidence, how do people deal with the possibility that some of the feedback is erroneous? The potential for error means that evidence evaluation must include decisions about when to “trust the data.” In this paper we present two studies that focus on subjects’ responses to erroneous feedback in a hypothesis testing situation—a variant of Wason’s (1960) 2–4–6 rule discovery task in which some feedback was subject tosystem error: “hits” were reported as “misses” and vice versa. Our results show that, in contrast to previous research, people are equally adept at identifying false negatives and false positives; further, successful subjects were less likely to use a positive test strategy (Klayman & Ha, 1987) than were unsuccessful subjects. Finally, although others have found that generating possible hypotheses prior to experimentation increases success and task efficiency, such a manipulation did little to mitigate the effects of system error.
Article PDF
Similar content being viewed by others
Avoid common mistakes on your manuscript.
References
]Brehmer, B. (1979). Preliminaries to a psychology of inference.Scandinavian Journal of Psychology,20, 193–210.
]Brehmer, B. (1980). In one word: Not from experience.Acta Psychologica,45, 223–241.
]Brehmer, B. (1987). Note on subjects’ hypotheses in multiple-cue probability learning.Organizational Behavior & Human Decision Processes,40, 323–329.
Bruner, J. S., Goodnow, J. J., &Austin, G. A. (1956).A study of thinking. New York: New York Science Editions.
Castellan, N. J. (1977). Decision making with multiple probabilistic cues. In N. J. Castellan, D. B. Pisoni, & G. R. Potts (Eds.),Cognitive theory (Vol. 2, pp. 117–147). Hillsdale, NJ: Erlbaum.
Chinn,C. A., &Brewer,W. F. (1993). Factors that influence how people respond to anomalous data. InProceedings of the Fifteenth Annual Conference of the Cognitive Science Society (pp. 318–323). Hillsdale, NJ: Erlbaum.
Doherty, M. E., &Tweney, R. D. (1988).The role of data and feedback error in inference and prediction. (Final report for ARI Contract MDA903-85-K-0193). Bowling Green, OH: Bowling Green State University.
]Einhorn, H. J., &Hogarth, R. M. (1981). Behavioral decision theory: Processes of judgment and choice.Annual Review of Psychology,32, 22–53.
Farris, H. H. (1992).Rule discovery heuristics: Goal-switching between counterfactual and positive test strategies in an adaptive system of heuristic search. Unpublished doctoral dissertation, University of California, Santa Barbara.
Freedman, E. G. (1992a, November).The effects of possible error and multiple hypotheses on scientific induction. Paper presented at the meeting of the Psychonomic Society, St. Louis.
Freedman, E. G. (1992b). Scientific induction: Multiple hypotheses and individual and group processes. InProceedings of the Fourteenth Annual Conference of the Cognitive Science Society (pp. 183–188). Hillsdale, NJ: Erlbaum.
]Gorman, M. E. (1986). How the possibility of error affects falsification on a task that models scientific problem solving.British Journal of Psychology,77, 85–96.
]Gorman, M. E. (1989). Error, falsification and scientific inference: An experimental investigation.Quarterly Journal of Experimental Psychology,41A, 385–412.
Gorman, M. E. (1992).Simulating science: Heuristics, mental models, and technoscientific thinking. Bloomington: Indiana University Press.
Kern, L. H. (1982).The effect of data error in inducing confirmatory inference strategies in scientific hypothesis testing. Unpublished doctoral dissertation, Ohio State University.
]Klahr, D., &Dunbar, K. (1988). Dual space search during scientific reasoning.Cognitive Science,12, 1–48.
]Klayman, J. (1984). Learning from feedback in probabilistic environments.Acta Psychologica,56, 81–92.
]Klayman, J. (1988). Cue discovery in probabilistic environments: Uncertainty and experimentation.Journal of Experimental Psychology: Learning, Memory, & Cognition,14, 317–330.
]Klayman, J., &Ha, Y.-W. (1987). Confirmation, disconfirmation, and information in hypothesis testing.Psychological Review,94, 211–228.
]Klayman, J., &Ha, Y.-W. (1989). Hypothesis testing in rule discovery: Strategy, structure, and content.Journal of Experimental Psychology: Learning, Memory, & Cognition,15, 596–604.
]Moertel, C. G., Fleming, T. R., Macdonald, J. S., Haller, D. G., Laurie, J. A., &Tangen, C. (1993). An evaluation of the carcinoembryonic antigen (CEA) test for monitoring patients with resected colon cancer.Journal of the American Medical Association,270, 943–948.
]O’Connor, R. M., Jr.,Doherty, M. E., &Tweney, R. D. (1989). The effects of system failure error on predictions.Organizational Behavior & Human Decision Processes,44, 1–11.
]Slovic, P., Fischhoff, B., &Lichtenstein, S. (1971). Behavioral decision theory.Annual Review of Psychology,28, 1–39.
Tweney, R. D., Doherty, M. E., &Mynatt, C. R. (1981). Hypothesis testing: The role of confirmation. In R. D. Tweney, M. E. Doherty, & C. R. Mynatt (Eds.),On scientific thinking (pp. 115–128). New York: Columbia University Press.
]Tweney, R. D., Doherty, M. E., Worner, W. J., Pliske, D. B., Mynatt, C. R., Gross, K. A., &Arkkelin, D. L. (1980). Strategies of rule discovery in an inference task.Quarterly Journal of Experimental Psychology,32, 109–123.
]Wason, P. C. (1960). On the failure to eliminate hypotheses in a conceptual task.Quarterly Journal of Experimental Psychology,12, 129–140.
Whether positive or negative, result of prostate cancer test can create a maze of questions. (1993, June 23).The New York Times, p. C12.
]York, K., Doherty, M., &Kamouri, J. (1987). The influence of cue unreliability on judgment in a multiple cue probability learning task.Organizational Behavior & Human Decision Processes,39, 303–317.
Author information
Authors and Affiliations
Corresponding author
Additional information
This research was originally conducted as part of the first author’s doctoral dissertation. Support was provided in part by an American Psychological Association Dissertation Research Award to the first author, and a grant to the second author from the National Institute of Child Health and Human Development (R01-HD25211).
Rights and permissions
About this article
Cite this article
Penner, D.E., Klahr, D. When to trust the data: Further investigations of system error in a scientific reasoning task. Mem Cogn 24, 655–668 (1996). https://doi.org/10.3758/BF03201090
Received:
Accepted:
Issue date:
DOI: https://doi.org/10.3758/BF03201090