Paul Wachtel, 10 March 2004

Some of you may have seen the article in the New York Times on Tuesday about the nuevo fanaticism when it comes to so-called "empirical validation" of psychological treatments. Being empirically responsible is a good thing, but when it gets confused with a tendentious definition of what empirical validation really is it becomes something quite else. For those of you to whom strictly randomized trials of DSM defined disorders with manuals as the only form of acceptable adherence check sounds like ideology disguised as science, you might enjoy the following suggestion below:

Parachute use to prevent death and major trauma related to gravitational challenge: systematic review of randomized controlled trials. Smith G.C. & Pell J.P., Department of Obstetrics and Gynecology, Cambridge University, Cambridge, CB2 - 2QQ, E-Mail <gcss2@cam.ac.uk>
OBJECTIVES: To determine whether parachutes are effective in preventing major trauma related to gravitational challenge.
DESIGN: Systematic review of randomized controlled trials.
DATA SOURCES: Medline, Web of Science, Embase, and the Cochrane Library databases; appropriate internet sites and citation lists.
STUDY SELECTION: Studies showing the effects of using a parachute during free fall.
MAIN OUTCOME MEASURE: Death or major trauma, defined as an injury severity score > 15.
RESULTS: We were unable to identify any randomized controlled trials of parachute intervention.
CONCLUSIONS: As with many interventions intended to prevent ill health, the effectiveness of parachutes has not been subjected to rigorous evaluation by using randomized controlled trials. Advocates of evidence based medicine have criticized the adoption of interventions evaluated by using only observational data. We think that everyone might benefit if the most radical protagonists of evidence based medicine organized and participated in a double blind, randomized, placebo controlled, crossover trial of the parachute.

Gerald Davison, 10 March 2004

Paul, with respect and affection, the analogy to parachutes is specious. As you know, I (and also with Marv Goldfried) have written on the limitations of randomized trials, so I don't swallow it whole. But observational data of deaths from parachutes not deploying is hardly like the "observational" reports that clinicians make of the outcomes (and processes) of their interventions. To quote some wag, the plural of "anecdote" is not data.

George Stricker, 10 March 2004

As luck would have it, the Times article earlier this week stimulated a conversation among my students about the topic, and I just sent the following note to them:

There is a wide set of conundrums associated with this whole controversy. All I can offer is my opinion, and others are entitled to theirs. First, if someone could give me a cookbook that said, if you do X, Y will happen, I would grab it in a minute. Many people would feel that such a cookbook would take all the glamour and excitement out of treatment, but we aren't (or shouldn't be) doing it for our own stimulation. Having said that, I also don't believe such a cookbook exists or is likely to, although there are some highly specific conditions that have some fairly well developed effective procedures (and not to use them because they aren't interesting is a gross disservice to the patient). Same with testing - if there was a true litmus test for any pathology or dynamics, I'd want it - I just don't know where to find it. Therefore, what we are thrown back on is to do the best we can, using whatever information we have. The twin dangers of this position is thinking we have more or less information than we do. One group, described in the article, suggest that anything that doesn't meet strict scientific scrutiny is terrible, overlooking that, held to that standard, we would do virtually nothing (as would physicians for most of what they do). The other group, recognizing the paucity of evidence, throw out any evidence that does exist, denying their patients the best care available because it isn't consistent with their prejudice, and dismissing all knowledge because knowledge is imperfect. That brings us back to Aristotle and the Golden Mean.

Gerald Davison, 10 March 2004

George, neither extreme is necessary or desirable. Even those who embrace treatment manuals are well aware of their limitations and of the need for idiographic analyses of specific cases. What I personally object to is President-Elect Levant espousing a public position against "too much science," or however he put it. I believe also that those who see no value in RCT's put forward an epistemology and specific operational suggestions on how to decide what we know or have some idea of what we may personally favor. Maybe it has to do with where one starts in deciding how to proceed with a patient. Where does one place his or her bets. What kind of data does one find persuasive, helpful, etc?

Tullio Carere-Comes, 10 March 2004

Hi Paul, thank you for calling our attention onto the New York Times Tuesday article. I have enjoyed the parachute analogy, and don't understand why Gerald Davison calls it specious. It simply reminds us that science is not always and not necessarily experimental. How could for instance historiography ever be experimental? Scientific methods should obviously be adapted to their objects, though this might not be obvious to the Randomized Clinical Trials' ideologists (to whom, as they often maintain, "there is only one science" - the experimental one, of course).

I agree with George's Golden Mean between clinical experience and intuition on one side, and scientific evidence on the other. Provided that we don't mistake RCT evidence for scientific evidence: it seems to me that there is not much science in RCT ideology. RCT ideology imagines that psychotherapy should be like any other medical treatment, i.e. specific disorders should be treated by means of (empirically supported) specific procedures. In this fantasy psychotherapy should be a bunch of short-term manualized treatments (ideally, one for every DSM-defined disorder). They should be short-term, because in genuine, open-ended psychotherapy the therapist is bound to change his/her approach at every step in every session, to meet his/her patient’s needs. RCT are about imaginary objects, not real therapy. Genuine therapy cannot be manualized, because it works with real, not standardized, people and conditions. Ergo, RCT evidence is of little use for real psychotherapy, while it brings with itself a significant risk of theoretical abuse. I too would object to Levant's position against "too much science": to me there is not too much science, there is only too much bad science.

Gerald Davison, 11 March 2004

We will have to agree to disagree on the aptness of Paul's parachute analogy, though one member of this listserv emailed me privately to suggest that Paul did not mean it seriously.

That aside, I would agree that RCT's are not the only method for concluding that we know something. An RCT, for example, gives only limited information on what to do at a specific point in time and space with a particular patient. To be sure, there are those favoring RCT's who might rightly be called ideologues, but I would respectfully suggest that this is a straw person and not worth our time. And calling people who see some value in RCT's ideologues is unproductive ad hominem and will not get us anywhere. Also, treatment manuals vary according to how "ballistic" they are as compared to responsive to the give-and-take of therapy interactions. I am familiar with many manuals and few if any of them distort reality in the way you are satirizing them.

So let me take the following tack: What epistemological criteria do you accept as defining something we assert we know in psychotherapy? Are the reports of clinicians of their experiences with patients enough for us? If so, how do we decide which reports have validity and/or heuristic value? Basically, what are the alternatives?

Paul Wachtel, 11 March 2004

Jerry, George, Tullio, and paratroopers, Jerry, to begin with, yes, the person who emailed you privately was correct, at least in large measure. I mainly forwarded the piece because I thought it was funny. But you are correct nonetheless to take it seriously, since I did also think it did a nice job of skewering something that needed skewering. So the question is: what is it that needs skewering and satire? It's not the need for science. I am in complete agreement that we need more science, more solid and carefully evaluated evidence, not less. If Ron Levant said that the problem is "too much science," then shame on him (I haven't read what he actually said). And I found just as delightful as the parachute study your quip about anecdote and data, which is similarly both funny and funny precisely because it is so on the mark. There is much too much "here's what I remember about what I felt about what my patient remembered about what he felt and besides, Kernberg said it so that's a cite to a data source."

But there is also a difference between science and ritual and between science and self-serving propaganda. My objection is in no way at all to the call for careful and systematic study in the most rigorous way that is suited to the phenomenon being studied. My problem with the narrow and tendentious way that empirical validation is often defined is that manuals are just one way of evaluating whether a treatment approach was followed and that manuals are only appropriate for evaluating whether a manualized treatment was being employed properly. By the requirement of manualization - and the misleading implication that that is the only way to check on treatment adherence - the very definition rules out as even possible to empirically evaluate any treatment that is not organized around a manual. (And by the way, the very use of manuals by some psychodynamic therapy researchers seems to me mainly a concession to the political and economic pressures to do so. Quite seriously, I only discovered years after they were published that several of the most prominent "treatment manuals" used in psychodynamic therapy research were manuals. I had read them, liked them, but thought they were "books." Treatment manuals that are "responsive to the give and take of therapy interactions" or the individuality of the patient are so largely because they are NOT really "manuals" except in the political sense that they "pass" and get the writer grants.

I also find tendentious the requirement, cited in many places, that treatment evaluations be limited to the treatment of a specific disorder. First of all, much of what we treat is only in a very limited sense specific disorders (there are some exceptions, to be sure, and for them I do think that the treatments that have been designed and evaluated for those disorders - often treatments that do lend themselves appropriately to manuals - should be the ones used). But for most of the DSM, I think the categories will look to psychologists looking back 100 years from now like the equivalent of earth, air, fire, and water, only with many more categories to satisfy the various constituencies.

So how should we proceed? In MANY ways. There should be a wide diversity of methods used to evaluate, each appropriate to the question and the phenomenon being pursued. This is NOT really difficult. Even randomized trials, for example, can be pursued by randomly assigning half the patients who come to the clinic to one treatment and half to another - WITHOUT categorizing them by whether they are earth, air, fire, or water - and see which does better with the general mix that comes to the clinic.

Should we STOP there? Absolutely not. The more specificity we can achieve the better. I'm in full agreement with that. If we can specify (the old saw) that particular treatments are better for particular patients, etc., obviously all the better. But if the only research that "COUNTS" - and the only research that is FUNDED - breaks the patients up into very narrow and specific DSM categories (and, in many instances, in addition, eliminates all the patients who have the "complications" and "confounding" features that just happen to be what plagues most people who go to therapists, then that is ideology and politics masquerading as science.

Similarly, concern with adherence, with whether the treatment administered is the one the researcher is claiming is the treatment does not require manuals. Indeed, even where manuals are appropriate, they are only as good as the ratings of whether the treatment has adhered to the manual. In exactly the same fashion, raters could evaluate, based on session samples, whether the treatment was truly a classical analysis, an object relations approach, a Gestalt approach, whatever.

OK, enough, enough. Life calls! Clearly Jerry, you stimulated me, as usual. Gotta get you to more SEPI meetings so we can talk about these things in person. There's only so long one can go on in email without carpal tunnel syndrome (for which I WOULD probably want a manualized treatment that has been thru rigorous RCTs).

Gerald Davison, 11 March 2004

Paul, as usual, well put, especially vis-à-vis the stranglehold that the DSM has on clinical research and grant-getting.

Hilde Rapp, 11 March 2004

Tullio, it may be worth noting that some of the fiercest critics of RCTs come from the pharmaceutical and medical constituency itself. One type of criticism relates to the incomparability of the demographic characteristics of the experimental sample to those of the target population that is meant to benefit from the intervention under test.

For example, basing judgments about the safety, appropriateness and dosage of drugs tested on white middle class college students, intended to benefit the elderly, children, or Asians with liver enzyme deficiencies can actually endanger their lives. Using RCT data from the nineteen seventies collected from 5000 or so middle class residents of beautiful Framingham in New England to standardize risk framing analysis score for heart disease has been shown to lead to wildly inaccurate predictions for expectable deaths in a British sample where these have been evaluated against follow up data from longitudinal studies (Welcome trust)

It may be salutary to remember that David Sackett, the 'father' of EBM defined evidence based medicine in 1996 as follows: 'the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients' (Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: what it is and what it isn't. Br. Med. J., 1996; 312:71-72). I'd sign that- wouldn't you?

Some criticisms relate to the observation that findings from some beautifully designed and meticulously carried out trials are less than useful due to poor construct validity issuing in asking the 'wrong questions', for instance when trials designed to support health policy decisions for improving the health of populations are misused for decision-making in the care of individual patients - or vice versa... Horses for courses!

The US Office of Technology and Assessment (1972) has been instrumental for the development of criteria for health technology assessment in the US, Canada, the UK etc which make very useful reading...

Tullio Carere-Comes, 11 March 2004

On 11-03-2004, "Gerald Davison" wrote:
So let me take the following tack: What epistemological criteria do you accept as defining something we assert we know in psychotherapy? Are the reports of clinicians of their experiences with patients enough for us? If so, how do we decide which reports have validity and/or heuristic value? Basically, what are the alternatives?

First premise: When I give a medicine, I get an effect that is only partially due to the specific action of the substance. A significant part of the effect, often the greatest part of it, is due to a non-specific action of the therapeutic intervention (placebo). Second premise: when I administer a psychotherapeutic procedure, I get an effect that is uncertainly related to the hypothesized specific action of the procedure, as the greater impact is due to the context in which the procedure is employed. To put it simply: patients do not respond to what I think I am doing, they respond to what THEY think is happening in the therapeutic interaction. The gap between what I think and what they think is much greater than in pharmacological treatments, to the point that in a meta-meta-analysis that compares different therapies the effect size appears to be almost irrelevant (Luborsky L., Rosenthal R., Diguer L., Andrusyna T.P., Berman J.S., Levitt J.T., Seligman D.A. & Krause E.D. (2002). The Dodo bird verdict is alive and well – mostly. Clinical Psychology: Science and Practice, 9, 1: 2-12. Commentaries [pp. 13-34]: D.L. Chambless; B.J. Rounsaville & K.M. Carrol; S. Messer & J. Wampold; K.J. Schneider; D.F. Klein; L.E. Beutler). A huge amount of research data supporting the second premise can be found in Wampold (The Great Psychotherapy Debate: Models, Methods and Findings. Mahwah, NJ: Lawrence Erlbaum. 2001). In other words, psychotherapy IS placebo, as someone has poignantly put it.

These premises do not make me jump to the conclusion that empirical research is useless. On the contrary it is useful, to the extent that it has corroborated what many of us suspected, i.e. that specific factors play a minor role in psychotherapy, compared to the common factors' major role. It can be even more useful, if it can help us look inside psychotherapy qua "placebo". Psychotherapy research should move in the very opposite direction than medical research: whereas the latter is only concerned with the objective activity of a therapeutic ingredient, and considers the subjective side (the placebo) as deserving little if any attention, the opposite is or should be true in the former.

How should it work? Let us consider for instance a basic constituent of "placebo", the common factor "secure base" or "secure attachment" (I maintain it is a common factor to the extent that no therapist could work without it, whatever his or her theoretical allegiance). It can be studied through an empirical research of the correlational (not experimental) type. For instance, at the end of a session of a real (not experimental or manualized) therapy both patient and therapist fill a questionnaire in which they rate on a 5-point scale the strength of the patient's need for secure base, and on another 5-point scale the quality of the therapist's response to that need. The concordance between patient's and therapist's ratings can be correlated with the session outcome, itself rated on a 5-point scale. Many other correlations like this can be studied. As a result of such correlational research, my heuristic intuition of the existence of a common factor called "secure base" or something like that, could be empirically corroborated (or falsified).

Unfortunately such a research (on which I am working, by the way) is unlikely to be funded or supported by public agencies, because it does not meet the criteria of experimental research that currently rule our field. Have I answered your question, Jerry?

Paolo Migone, 11 March 2004

Regarding the problem of the importance of science, RCT, Empirically Supported Treatments (EST) etc., I would like to say that in a future issue of Psychological Bulletin, coming out in May 2994, there will be a very important article by Drew Westen, Kate Morrison, and Heather Thompson-Brenner, titled "The empirical status of empirically supported psychotherapies: assumptions, findings, and reporting in controlled clinical trials", which is a detailed critique to the EST ideology. In my opinion this paper will be a point of reference for all interested in this issue (Drew Westen sent me this paper that he begun about 5 years ago, and I am trying to obtain the right for the Italian translation). What I find interesting in this article is that Westen et al. are not at all against EST, or against science, or against the need and importance of rigorous outcome research. On the contrary, they strongly believe in science, and is just for this reason that they criticize some shortcoming of one kind of psychotherapy research, in order to "improve" the field of research. And, most interestingly, they use only empirical data coming from the same source of evidence of EST, in order to contradict some aspects of EST ideology and background. For example, the various problems mentioned by Paul Wachtel (such as the paradox of psychotherapy research, i.e., the more "realiable" and well done is a study, it loses "validity", etc.) are examined in detailed by Westen et al.

In other words, I do not think at all that there is a dichotomy between science or something else (intuition, art, or whatsoever), but the dichotomy is only between different kinds of methodologies of scientific research, that can be more or less sophisticated.

Mardi Horowitz, 12 March 2004

Sorry, I did not read all the empirical e-mails so someone may have made this point. I like the direction of the discussion, it bifurcates well into a scientific dialogue and a political one. As to the former, I think we can draw the line at clinical reports of only one case. While I do that all the time, I would not judge the results empirical, UNLESS something else happens: other clinicians have to nod in agreement, as in now that you formulate it thus and so I concur from my own experience. We have very few papers where anyone tries to see what percent agree, how much, on what basis. I think that is the fuzzy edge, maybe and maybe not yet "empirical". Certainly, however, there is excessive emphasis on the controlled clinical trial. In my opinion that is the last thing to do, after a bunch of much less expensive and limited use of correlational, multiple regression (step wise), and descriptive plus reliability of observations made type studies: only then can the multiple interactive variables be examined. A controlled trial is the last confirmatory step, and they are too often the only thing funded, and too constrained to an already outmoded diagnostic system, whose categorical disorders are not homogeneous in causation or course.

Gerald Davison, 12 March 2004

Well said!

Tullio Carere-Comes, 12 March 2004

Thank you Hilde for your data. I would add that it has been shown (N Engl J Med, 2000, 342) that if observational studies are well designed, their results are not very different from those of well designed RCTs on the same matter, but produce more data that RCTs cannot record. I am not against RCTs in principle, I am against the dominant position they have received in both medical and psychological research, one that by far exceeds their usefulness. We should get rid of the deleterious Popperian philosophy, according to which observation is only good for generating hypotheses to put to empirical test. Well disciplined observation is a source of valuable data in its own right. In some cases it may be worth testing these data through RCTs, but in psychotherapy research I believe that it is much less often the case than funding agencies seem to believe.

Tyler Carpenter, 12 March 2004

Working effectively in a prison setting has made me even more of a pragmatists than I was before I started there. From my point of view empirically supported treatments (or whatever they're called - "evidence based practice" is the buzz word in correctional work I think), provide manuals, protocols that go beyond what a small group of more highly and expensively trained personnel could provide and give a framework for understanding a lot more about the patient's symptoms and potential trajectories than I might otherwise given the brief time available to work. I then see therapy as working at the interstices or nexus of the systems from inside to outside the inmate/patient. This involves leveraging what can be brought to bear in the usual therapeutic task and may involve consulting to corrections personnel, getting a med consult to tweak a symptom dimension, working on the dynamic aspects evoked by the use of whatever treatment philosophy is brought to bear.

Hilde Rapp, 12 March 2004

Dear Tullio, I entirely agree with you, and I have just written something where I make a sustained plea for reinstating observation at the center of our work so that re-'search' becomes a more systematized aspect of our general 'search' for truth..

Franz Caspar, 12 March 2004

Dear Colleagues, I sure do not want to reiterate well formulated points in the recent exchange. Just a general comment. We need more awareness of what can be said and what cannot be said based on a particular study or type of study. The logic of RCTs requires the precise definition of what the treatment was (which is usually done by manuals which would have to be rather narrow to achieve what their job is – I had similar experiences as Paul with having a hard time seeing "books" as "manuals". In principle, the definition of what the treatment was could also be achieved by prescribing heuristic rules and then precisely describing the actual procedure after termination!). Logically, RCTs have the highest value in terms of supporting causal conclusions. Emphasizing the limits of RCTs can not do away with this! We will need additional RCTs in the future to answer some relevant questions, while it is obvious that other questions (and some are the most relevant for practitioners!) need other types of research. Different types of research can't replace each other but are rather in a complementary relationship. We need effectiveness research, research beyond narrow DSM categories, research on effective principles, implementation research, service research, research on therapists and how they can or can't use empirical evidence, etc. Looking into the discussions, e.g. in APA divisions, or NIMH, I find a lot of awareness for this, although the consequences have not been drawn sufficiently. Research and instruction of practitioners costs money, doing all kind of complementary types of research costs even more money. So we also need some realism and patience. In the meantime we need to acknowledge the gaps in the empirical basis for deriving our therapeutic action from high quality empirical evidence, but to strive for keeping these gaps as small as possible.

This is a practical sampling issue. Doesn't one run into the same problem with observational studies? Issues of external validity are present in all knowledge-gathering enterprises. It may be salutary to remember that David Sackett, the 'father' of EBM defined evidence based medicine in 1996 as follows: 'the conscientious, explicit and judicious use of current best evidence in making decisions about the care of individual patients' The issue is how one defines "evidence." Aye, there's the rub.

I also agree very much with your observation that we have much to learn from academic psychology especially developmental psychology and psychopathology. I wonder whether you might have some relevant writings to 'throw ' into the integrative pot? I am happy for you to use anything I may have contributed to the discussion and, by way of seed grain for references, here is the reference for the Sackett quote: Sackett D, Rosenberg W, Gray J, Haynes R, Richardson W. Evidence based medicine: what it is and what it isn't. Br. Med. J. 1996; 312:71-72

Dear Tullio, I think that the concept of dialectics that Popper "hated" has nothing to do with the meaning of dialectics that we use here: here we do not talk of the philosophical meaning of dialectics, but simply a kind of scientific research in which we perform not only "bottom-up" experimental studies but also a sort of "top-down" research, in order to arrive quicker at meaningful discoveries. Rather than the term "dialectic", a better term here would be "synergic".

Dear Tullio, regarding your well known emphasis on the importance of what you call "dialectics" in psychotherapy, I came across this article: Monroe Pray, "The classical-relational schism and psychic conflict". Journal of the American Psychoanalytic Association, 2002, 50, 1: 249-280.

Tullio, I'd like to suggest that both the use of the terms medicalization and chaos have rhetorical implications that are neither integrative nor indicative of either real science or whatever it is some call what we do (personally I'm always a practitioner-scientist whatever someone chooses to make of it). It seems to me I remember a marvelous discussion with Bernie Beitman at one of our conferences when he mentioned to me the link between the patient's psychology and the therapeutic action of the medication. It seems to me whether we are talking mind-body immune function and antibiotic action on disease process or SSRI or mood stabilizer or atypical tranquilizer and treatment of violent criminals in a number of contexts simultaneously, it's all change and all integrative when done attentively. The question of science then, at least in terms of shared overarching frameworks of comprehensive and valid constructs, becomes a matter of education and creative discussion rather than science vs. art.

Psychotherapy Integration Papers

Influential Readings on Psychotherapy Integration

CyberPsych Site Index

next

On Psychotherapy Research

(SEPI Forum, March-April 2004)

Paul Wachtel, 10 March 2004

Gerald Davison, 10 March 2004

George Stricker, 10 March 2004