In case you missed it, the Journal of Personality and Social Psychology, a flagship APA journal, published a study by Daryl Bem containing evidence for psi (precognition). I didn't really want to post in too much detail about this study (which has been doing the rounds online all week) because I'm not that interested in being a science journalist. But I did feel it was worth posting a few links and some brief comments to this provocative paper, because it raises a lot of interesting questions about the business of doing psychological science.Should a paper on precognition be published in a major social psychology journal? How did Bem get his results? How seriously should we take what seems to be evidence for something that a lot of other science suggests is impossible?
Bem's paper contains 9 studies, each of which takes a classic effect (e.g. priming) and reverses the order in which things were presented, so that the stimulus we know affects behaviour came after the response had been made. Bem's logic was that if you could reliably see the known effect under these conditions, this would count as evidence that people's behaviour was being affected by precognitive knowledge of the future event. He got small but statistically significant effects in this direction in each of the 9 studies.
- I don't believe a word of it because a) let's face it, it's about precognition and b) there's simply no effort to propose a mechanism that might support such an outrageous claim (and no, 'quantum mechanics' is not a mechanism). Bem explicitly states that coming up with a mechanism isn't his job and he's just 'reporting the data'. But this is precisely the problem with psychology right now - not enough theory - and the links below that talk about the analysis problems with this paper (and all statistical testing in the social sciences) make good points about the fact that statistical testing in the absence of a clear theory which includes mechanisms is effectively a fishing trip. There's also a nice discussion here about why it's healthy to be immediately sceptical of a study that claims to have found something inconsistent with the rest of science.
- That said, it is a rigourous paper; there are 9 carefully run studies with a lot of detail, and the goal (to provide a framework for replication and testing of psi) is admirable. It might be wrong, but it's at least replicable and falisfiable, and I can respect that. I'm also not grumpy with JPSP for publishing it, because the paper has done what it was intended to do and sparked a lot of discussion, analysis and replication. The risk, of course, is that people will only remember the original, positive study and never hear about the failures to replicate. I hope the various science bloggers talking about this study remember to follow up.
Speaking of following up: there are already three failures to replicate that I know of, here (and they have more coming), here and here; if you are planning a replication, register it on Richard Wiseman's site here. There has been one detailed analysis of the statistical problems that are probably leading to the significant results, and this post led me to this book preview
by two authors describing in great detail the problem with the kind of
significance testing we do in the social sciences. Maggie Koerth-Baker has been keeping tabs on all this on her Twitter feed.
References
Bem, D (in press). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology. Download.
The
real lesson? This is the level of methodological scrutiny every paper
should receive, and not just the ones you think are crazy: the ones you
like and rely on for your own work should get a good working over like
this too (especially these ones; and I'm as guilty on this as everyone
else). In the social sciences we depend entirely on statistics to make our points; this dependence on distributions makes replication an absolute necessity. I just hope this is the lesson we learn, because it's not rewarded much right now.
Update, 2010.11.18: Richard Wiseman thinks he has a methodological flaw.
Update, 2010.11.18: Richard Wiseman thinks he has a methodological flaw.
References
Bem, D (in press). Feeling the future: Experimental evidence for anomalous retroactive influences on cognition and affect. Journal of Personality and Social Psychology. Download.
Posts
"I don't believe a word of it because a) let's face it, it's about precognition"
ReplyDeleteLOL! Nice confirmation bias!
@Crispin: Not totally sure what the LOL is about. When a hypothesis seems extremely unlikely ex ante, even a highly rigorous study shouldn't lead one to suddenly start believing it. Isn't that what Bayes & his Law are all about?
ReplyDeleteIts funny how Bayes law is only invoked by most researchers when there are frequentist findings which seem "impossible".
ReplyDeleteIn any case the Wagenmacher paper above treats Bem's experiment as if it were the only evidence, when in fact there have been studies conducted on this for over one hundred years. Any true Bayesian account would need to look at all these studies and incorporate them into the prior before looking at the most recent evidence.
Seems like a worthwhile endeavour for an amateur statistician and psychologist, maybe i'll do it over my holidays....
You have to be careful about comparing apples with oranges; the point of Bem's paper was to try and create a framework for replication, something the previous work had always lacked. You can't just throw any old thing into the prior (well you can, but it's an error :)
ReplyDeleteBayesian stats is actually a much more robust framework for analysis. John Krushke from IU has had a couple of papers laying this out recently (eg this one) and he's right.
Your response implies an ignorance of this field. The framework Bem works in is essentially the same as that used by many other successful studies of highly similar designs. He references some of them, and many more could be cited. There is a sizable background of quite comparable work.
Deletehttp://improbable.com/airchives/paperair/volume1/v1i3/air-1-3-apples.html
ReplyDelete"This is the level of methodological scrutiny every paper should receive, and not just the ones you think are crazy: the ones you like and rely on for your own work should get a good working over like this too (especially these ones; and I'm as guilty on this as everyone else)."
ReplyDeleteNot necessarily. Suppose I do a study finding that alcohol impairs performance on a memory task.
Now, given that we have vast amounts of prior data showing that alcohol does in fact do that, this result is probably correct. Of course I might have made a serious mistake even so, and ideally one would check every study in great detail. I think with the rise of research blogging, this is happening more and more often, which is a great thing. But given that it takes time, we need to prioritize, and studies which fit with lots of other data are low on the list.
Now you might well ask what the point of such a study is, if we all know the result, but that's a separate issue.
Four questions pop into my mind right away:
ReplyDelete1. How much alcohol?
2. What counts as impairment?
3. What memory task?
Now you are likely to have perfectly acceptable operational definitions for everything, which is fine, but it then gives rise to the fourth, key question:
4. How do these definitions relate to the previous work?
This question is vital and unavoidable in the social sciences, where methods and techniques vary in ways they don't for, say, a standard experimental technique in chemistry. Critically, the question even remains for common effects, and so should really get asked every time, all the time.
Psi doesn't exist. WHY? The answer comes from metaphysics. The following page must give answer to the nature of existence:
ReplyDelete"Metaphysics : Importance of Philosophy"
http://www.importanceofphilosophy.com/Metaphysics_Main.html
Just curious, why are psychologists wasting time & money researching this nonsense? Is it because they lack philosophical reasoning background or they do it because they believe psychics?
Not all psychologists are doing this, and like with Freud's work, many are doing research just to prove Bem wrong.
DeleteEither way, I'm not sure why every idea isn't worth looking at. (Of course the world is flat, don't be outrageous!)
Because we have reason to believe some ideas are wrong: see modern physics on time travel
DeleteAnonymous,
DeleteEvery idea is not worth looking at - if by 'worth' you mean that the cost and the likely outcome balance favorably. That is what the word means, right? --- "Is it worth it to buy the new BMW?" "Is it worth it to fund research into hypothesis X?" --- Same question.
On the other hand, as I have argued elsewhere, there is no particular need to be critical of Bem. He is at the stage of life and career where he can research what he wants, and if he chooses to research something I think is silly, so be it. The big questions this incident raises are questions about how our discipline should operate. Certainly you would agree that, as a discipline, we should not grant equal resources towards answering all questions. Wouldn't you?
The Metaphysics Link in no way argues against PSI.
Deletecongratulations, you win the award for the weakest post in this thread!
ReplyDeletePhysics Nobel Laureate, late Richard Feynman gave a lecture at Caltech in the 1970s addressing this kind of nonsense research that Daryl Bem is doing.
ReplyDeleteCARGO CULT SCIENCE
Go ahead and read it Gav, because it may enlightened you about science.
Metaphysics isn't a final arbiter of anything; it's simply the set of things you take to be true, for whatever reason (logic, empirical data, etc).
ReplyDeleteJust curious, why are psychologists wasting time & money researching this nonsense? Is it because they lack philosophical reasoning background or they do it because they believe psychics?
I can't believe I'm defending Bem here, but the answer is simply 'any question asked properly will produce an answer of value'. Psi, etc is a topic because it's been suggested for a long time that we might have such abilities; this paper, whatever it's problems, has asked the question carefully enough to allow rigorous external examination. Whether or not it stands up to that exam is a separate issue.
Do I think it was worth running the study? No, not really. But it's produced a lot of good critical analysis and, as we keep telling governments and funding agencies, you can't be too top-down and prescriptive about what scientists get up to because it's hard to see where it will lead. So the fact I think it's a bit of a waste of time shouldn't have stopped Bem from running careful studies, because I don't want his lack of interest in perception/action to stop me from running my careful studies.
Andrew, you can try and spin all you like, but there is no other way to refute the nonsense that Daryl Bem has published other than metaphysical arguments?
ReplyDeleteHow else are you going to refute it? Statistical analysis? Look statistics has got nothing to do with physical reality. Sure, its a useful tool, but it cannot dictate to what existence should be. One can design an experiment to test if God exist or not, just based on gut feeling, hunching, etc,... and using the same statistical hypothesis testing that Dr Bem's used. Guess what? The tests would probably showed that there is statistical significance that God exist. You have to ask first? On what physics (or physical grounds) that such experimental design is based on other than a statistical hypothesis test? Can you see how such arbitrariness should be dismissed with no ifs no buts? Metaphysics gives us that guidance.
Dr Bem's research was not based on a physical concept of reality. It is arbitrary and therefore must be thrown out, no ifs no buts. It violates causality, it is unphysical (ie, doesn't exist), and logical consequence from that is that psi must not or cannot exist or otherwise, it violates the laws of physics and if anyone has to accept that laws of physics can be frequently violated then that, reduces us (conscious observers on this universe) and everything to mere illusion.
Physics is build on philosophy (metaphysics), and human mind (brain) is made of atoms/molecules/polymers, etc, and therefore whatever actions of its neurons the law of physics must still be applied to those physical components. There is no escaping from this fact.
When metaphysics is not used as a guide to physics, then you have all hokum claims being proposed, such as mind over matter, out of body experience, precognition and some even claim that quantum mechanics confirmed those things. But they only deluding themselves.
For psi to exist it must be non-physical, ie, it violates causality (laws of physics). The other word for non-physical is non-existence.
The Bem study is a waste of time, period. Don't even suggest that it warranted government funding, and also don't suggest that his work is science. It is not science. It is a disgrace to real science to suggest that Bem's work is science.
Now that was the reply post I was hoping for the first time round!
ReplyDeleteLots of things to think about if results aren't made up.
ReplyDeleteIn the experiments where future experience impacts the past (e.g. priming,recall), why didn't he have all the pictures displayed on the computer and then have the view of the subjects blocked -- this would rule out any possible artifacts of computer display.
He also could have had many more controls. For example, in the erotic stimuli experiment, he could have had the subjects choose the curtain, and then have the pictures displayed on the computer, but in 1/2 the cases the subjects' view could be blocked and 1/2 the cases the subject's view could not be blocked.
In the experiment on priming, he should have neutral as well as incongruent and congruent pictures.
If the subjects are perceiving the future, then what happens if you change the future displays based upon the subjects responses -- wouldn't that automatically negate or even reverse the effect of precognition?
Basically, if I were him and sitting on the biggest experiment in history, I would have been a lot more careful and a lot more thoughtful.
Actually, for all it's sins, the psi study is pretty scientific. It was hypothesis driven, carefully controlled, carefully documented for future replication by independent labs, the analyses (while probably problematic) were transparent and standard.
ReplyDeleteThere's a difference between 'scientific' and 'true'. The former is a method and you can apply it to nearly any question (whether God exists is a shady one, because of the problems in identifying what might count as evidence ahead of time; Bem doesn't have this problem though). I don't think that psi exists because of what I understand about the world through science as it currently stands; but the thing I enjoy most about being a scientist is the permission it gives me to consider the possibility I might be wrong.
There are a lot of quantum physicists pleased you aren't in charge of funding decisions, by the way. While not an explanation for psi, it's a depressingly accurate but entirely bizarre description of what things might really be like, and it violates every rule of Newtonian mechanics. Metaphysics exists, like all these things, in an historical context, and your approach rules out the possibility of finding out something truly weird yet true.
Running experiments that fly in the face of the received wisdom, but that are still disciplined and interpretable, is the single great contribution of the scientific method to the search for greater understanding. It's how we make progress.
I think psi is a rubbish idea, but the onus is now on us to defend this received wisdom without cheating. I say good times! Bring it on :)
@Anonymous: find yourself a computer and a subject pool and knock yourself out. Those are all perfectly sensible things to check, but they aren't a critique unless you can show they really did matter. Good luck! There'll be plenty of interest in the results by the looks of things.
ReplyDeleteAndrew, you're starting to talk like a psuedo-science advocate as Feynman addressed in his Caltech speech.
ReplyDeleteYou said...
Actually, for all it's sins, the psi study is pretty scientific.
Now, I am amazed that you call yourself a scientist with that sort of comment you made above. Where do you draw the line of what is being regarded as science and what is not science? Psi is not scientific. It is not grounded in any physical theory or anything like that? Which branch of physics that psi theory falls in? None whatsoever. It is arbitrary and has no basis in reality (either observational or theoretical framework). You must throw out arbitrary assertions or otherwise, one would have to test an infinite set of hypotheses that one can just dream up. That's why we built hadron colliders and things like that because the theory guides us to search for such possibility since theory says that such phenomena in nature exists, ie, the equations and their solutions tell us some hints about those yet unobservables. Physical laws are very well hidden. We cannot run an infinite sets of experiments to detect hadrons just because we think such particle or force x, y, z do exist. There must be some foundations to start with.
Psi research has no foundations at all. It is similar to homeopathy. No foundation in reality to start with. They're arbitrary and must be dismissed as metaphysics teaches us.
It is a waste of time to prove psi. Metaphysics says, that the onus to prove are those that make the claim, ie, Daryl Bem, but not us (the opponents) as you stated. His statistical correlation is not proof. Psychologists don't develop theory of nature or physical reality. You use statistical hypothesis testing, which is descriptive in nature and unreliable. The laws of the physical world were not invented via statistical hypothesis testing.
It is time that you start saying that psi research is nonsense and non-scientific and stop trying to argue that it is scientific.
Falafulu, your comment got flagged as spam by Blogger - sorry!
ReplyDeleteI can't imagine why...
ReplyDeleteYour primary arguments against the study provide no illumination I'm afraid.
ReplyDelete"I don't believe a word of it because a) let's face it, it's about precognition."
The reason not to believe it because no one could possibly believe it. Circular reasoning, to say the least.
"and b) there's simply no effort to propose a mechanism that might support such an outrageous claim (and no, 'quantum mechanics' is not a mechanism)."
The study is quite clear about the absence of mechanisms. Bem is forthright and illuminating on why data must speak for themselves, even in the absence of theory. They have done so with respect to some of the most important scientific theories. Quantum mechanics is one such possibility, but he does not stake the study on that proposal.
The one failed replication thus far used participants recruited over the internet who participated online. This is a critical difference.
Wagenmakers et al.s' article would invalidate most social science research, not just Bem's study. I would challenge Wagenmaker et al. to identify a single major study in the psychological literature that uses a Bayesian T-test. It doesn't happen. Their rejoinder is fatally tainted as well by efforts to use Bem's previous words on conducting research to condemn this study. “There are two possible articles you can write: (1) the article you planned to write when you designed your study or (2) the article that makes the most sense now that you have seen the results. They are rarely the same, and the correct answer is (2).”(Bem, 2003, pp. 171-172). Anyone who has done serious psychological research knows that Bem is exactly right about that. No study is ever purely hypothesis-driven or explorative. Methods are ALWAYS written before the introduction. Why? For exactly, the reasons Bem cites.
Wagenmakers article also makes much of the roulette wheel comparison. I think this point is quite legitimate, and must be taken seriously. However, there are critical differences. Bem deliberately used paradigms that employed implicit methods, because he did not want people trying to predict the future. This is exactly what roulette players are trying to do. In addition, he used stimuli that would have evolutionary relevance and survival value. Needless to say, ancestral humans were not playing roulette. But they were avoiding violence and seeking sex.
I believe data. And so far Bem has produced some extraordinarily compelling data in an exceptionally well-designed study, a model of scholarship. Whether his results will be borne out remains to be seen. I await the answer without an investment in the outcome either way.
Hi Anthony - thanks for your comments!
ReplyDeleteA couple of things: my points aren't arguments, not even a little bit. I had no intention of those being convincing, they were just my take. The reason is that I have limited time and I'm not interested in using it to chase psi researchers. But people who are more invested have been, I think, very measured and precise in their critiques, so I link to them :)
Let me add another link which the author emailed to me; it expands my basic hunch into an argument you might find compelling.
On mechanism: Sabrina and I have been talking about this more too. One of the reasons we started this blog was to explore actual theories of behaviour because psychology suffers horribly from a lack of good theory. Data are, indeed, wonderful; but the first thing you learn on philosophy of science is that pure induction from data is impossible. Every data point has theoretical assumptions built into it so it's up to you to front up to these explicitly.
Bem's lack of mechanism is a serious problem, as that link talks about, because he's proposing something that literally makes no sense given much of the rest of science. Given this, and given the analysis concerns, his wonderful data may simply be a fishing trip that got lucky for no reason other than everyone gets lucky eventually.
On the statistics side: saying that the Wagenmakers et al paper can be ignored because if true it hurts all social science isn't a argument. They are, in fact, correct: GLM stats and p values have been horribly abused by us for years as our standard practice. It is, in fact, a worry.
My take is still, though, that I'm pleased Bem threw modern rigourous methods and analyses at this problem and found what he did, because maybe, just maybe, we'll get more serious about the problems that let him find those results.
Even Alan Turing didn't exclude the possibility of ESP (c.f. (9) The Argument from Extra-Sensory Perception in "Computing machinery and intelligence" http://www.abelard.org/turpap/turpap.php )
ReplyDeleteI'm sorry for Your Kuhnian paradigm, "Mr. Falafulu Fisi" but if I would have to choose between doctrinal approach You call "science" and intuition of Turing's genius, I would definitiely choose the latter one.
Congratulations to dr. Daryl Bem! ESP is known from long times and must be studied seriously. Certainly Copernicus, Newton and Einstein, as well as Freud and Jung may have their ideas criticized the same way Bem's studies are being criticized now. It is time to break some old paradigms, as it was already advocated in Fritjoff Capra's works like " The Tao of Physics" and "The Turning Point". Nobel Prize to Daryl Bem !!
ReplyDeleteIf your a skeptic than don't even bother commenting. Anyway I've been having precognition incedents lately like joking about what might happen and then it actually ends up happening or i'd think about something then it would happen a few seconds later. Now for telepathy I read about how theres a sender and a receiver and I've been practicing with my friends and I'm really good at sending thoughts and kind of good at recieving them but is there a better way to develop this ability?
ReplyDeleteBrad, if you want some answers, you might be interested in Richard Wiseman's book Paranormality.
ReplyDeleteThe day we let theory trump observation is the day science dies.
ReplyDeleteNice post, and more balanced and sensible than most that I've seen.
ReplyDeleteRandom commentary:
1. If the only effect of this paper is to cause people to become much more careful about leaping to conclusions on the basis of spurious significance (especially when the number of comparisons is not predetermined and the significance threshold is not adjusted accordingly), Bem's paper will be the most valuable contribution to the literature in decades.
2. It's lovely to have an explanatory theory, but the observations come first. If Bem's results are really repeatable, then we're going to have to deal with it. If they aren't, then there's nothing to see here -- move along.
3. There is nothing at all wrong with exploratory data analysis finding hypotheses that you didn't start with. Any hypothesis you come out with at the end is falsifiable. The key is to then independently test that hypothesis in isolation, and see if it's repeatable. It's never science if you don't get that far.
4. I don't buy the arguments that extraordinary claims require extraordinary evidence, because it is possible that the only thing that makes them 'extraordinary' is entrenched error. The theory of continental drift was laughed out of the room for decades (centuries?), because no plausible mechanism could be stated and "everyone knew" that continents couldn't wander about. The _data_, properly interpreted, were incontrovertible even then. That we didn't accept the data was mere prejudice and pigheadedness.
"there's simply no effort to propose a mechanism that might support such an outrageous claim"
ReplyDeleteWhile I don't buy it myself, this is just ridiculous. Bem supposedly proved an effect, its cause is irrelevant to wether it is real or not.
By following your logic we should ignore facts for which we don't already have an explanation.
Bem is proposing something that requires a violation of the known laws of physics. He owes us a swing at a real mechanism. "We Report, You Decide" isn't quite how science works.
ReplyDeleteI know I am stating the bloody obvious and that it has been said on this site elsewhere, but:
ReplyDelete1) Quantum mechanics can never be an explanation for acausal effects, precisely because it is causal itself.
2) An acceptable paper that breaks with known physics does as the recent OPERA paper does: prints the results, tells us all that they have done to check for the problem, and asks for people surveying what went wrong. Only if it can survive that and then be repeated, it can be an actual effect.
"there's simply no effort to propose a mechanism that might support such an outrageous claim"
ReplyDeleteHere's a paper from 2006 by Russel Targ, well-known physicist and "remote viewing" expert, in which he proposes a mechanism to support precognition ("Minkowski manifolds"). Targ is not cited directly in Bem's references.
https://netfiles.umn.edu/xythoswfs/webui/_xy-20376224_1-t_chSse0i4
What would be another cool name for the sixth sense or precognition sense?
ReplyDeleteFor the falulu fellow (I know, it's spelled wrong on purpose - let's have a little sympathy for a fellow who can't reason):
ReplyDeleteNotes on Materialism
Consequent to a recent argument about Materialism in which I expressed myself badly, sometimes incoherently, made mistakes, became very irritable, was occasionally rude and failed to make any headway whatsoever, for all of which I can only apologise sincerely to the participants, it seemed a good idea to write an essay about Materialism, so that if I am daft enough to get into another such argument I can just point to it and know that it is reasonably clear and expresses my view without prevarication, so that its defenders can be clear about the charges I would lay at their door.
Materialism is not usually considered a crime, and often it is not. When professional academics argue for this useless and life-denying doctrine, however, with all their power to influence the minds of young people, I believe that this is a crime against humanity. Professionals are paid to think clearly and honestly about such issues, on our behalf, and to truthfully report back with their findings. This is not at all what many professionals do. Nothing like it. Many publicly endorse Materialism, even though for anyone capable of passing an undergraduate degree a little thought will show that Materialism is logically indefensible, utterly useless, and not even as plausible as Idealism. For such a person Materialism is not merely a mistake, it is a failure to think through the issues and thus earn their salary. This is unprofessional conduct. If such a person then chooses to poison the minds of their students by confusing them with their eloquence into thinking that Materialism is a rational theory, then this would be a matter for the ethics committee.
It is not difficult to understand why many people are materialists, and I must stress that these strong views concern only professional academics, not all materialists. I was a materialist on and off for many years, simply because I could not think of a better idea. But its faults and weaknesses are not obscure, I could see them as a teenager, and there is not one philosopher, not one person, even among its advocates, who can make any sense of it. This can be said with confidence since it is a theory that contradicts reason. Materialists should find this no more difficult to verify than their opponents.
One astonishing feature of the debate about Materialism is that so many of the scientists who reject metaphysics as a source of truth, as we must do for Materialism, nevertheless imagine that they have the right to promote a metaphysical conjecture in a language which suggests that only a fool could fail to see that it must be true. This is an abandonment of the scientific method, a rejection of rationality and reason, and it is foolish beyond words. It is certainly dishonest. An endorsement of Materialism could never have anything to do with any scientific evidence. It is not a scientific theory but a metaphysical conjecture that floats entirely free from physics. It is simply a well-known fact that the scientific evidence has nothing to say about whether Materialism, the idea that matter is fundamental, is any more plausible than Idealism, the idea that mind is fundamental, and could never even be enough to falsify it. When a person says that they are a materialist or idealist they are not doing physics but metaphysics, whether they want to do it or not. By claiming that Materialism or Idealism would be the correct solution for one of metaphysics’ most ancient and venerable logical dilemmas, one that remains a logical dilemma to this day, they are doing it very badly indeed. In metaphysics, the discipline whose responsibility it is to decide such questions, both these ideas fail. This can be demonstrated. It is not a matter of opinion. The idea that Materialism would be a rational metaphysical position to endorse is demonstrably irrational.
if you are interested in the entire article, please write me at donsalmon7@gmail.com
ReplyDeleteOh, and it was written by Peter Jones.
ReplyDelete