Wednesday, June 6, 2001
The spin cycle began immediately the day in May the U.S. Census Bureau released new data about American households: There were 25% more single mothers than a decade ago; 62% more single fathers; and 72% more unmarried partners.
High-profile conservative Bill Bennett appeared on TV and decried the continued decline of the traditional nuclear family. He cited other research he said proved the superiority of intact, two-parent families for children. Dan Quayle prepared a column saying he was still right.
At the same time, Nancy Folbre, a University of Massachusetts researcher, received a call from the Institute for Public Accuracy, a small liberal nonprofit group, asking her to help craft a press release with a different interpretation. Folbre, a self-described feminist progressive economist, feared the conservatives' agenda would hurt children by replacing poverty programs with pro-marriage policies. She jumped at the chance to "get in a lick."
That day, the IPA e-mailed a press release to the nation's journalists saying many mothers and fathers stay involved with their children even if they aren't counted among traditional married family households. It quoted Folbre: "We shouldn't stigmatize single-parent families just because they don't conform to some traditional stereotype, or because they violate the personal, ethical or religious standards of influential members of our society."
Her subsequent appearance on four talk shows didn't match the reach of Bennett or Quayle. But Folbre was satisfied. At least she had lobbed a hand grenade into the nation's political and ideological battle over the meaning of social statistics.
"From my point of view," she said, "it's like guerrilla warfare."
Anyone who harbors an idea that researchers produce Olympian truths about social issues is naive, the researchers themselves say. "All studies are possibly wrong, whether they're surveys, clinical trials, or whatever," said Tom Smith, director of the General Social Survey at the University of Chicago's National Opinion Research Center.
Not only are survey results such as the census open to interpretation by rival advocates who hope to influence laws, judges or public opinion; research itself is also vulnerable to manipulation by individuals hoping to win grants or promote their own careers. Researchers say they have been under increasing pressure in recent years to produce results that are either politically correct or can tell an exciting story in a few sound bites.
Though honest researchers stop short of cooking results to prove a particular theory, they shape their studies nevertheless through countless choices from what questions they ask to how they define terms such as "child abuse" or "sexual assault" or present their results.
The tide of bad statistics is rising, said David Murray, director of the Washington, D.C.-based Statistical Assessment Service. "It's everywhere now. In politics. In polling. The interpretation of public reaction. Physical sciences. It's the way you win arguments. The way you make points."
Because our culture grants so much authority to statistics in social debates, the bad ones have great potential to mislead, he said. People use statistics in the way priests use mysterious language. "The language of science invokes higher authority. We speak Latin. Egyptian. You believe us."
What's more, only a handful of scientists are fully trained in complex statistical methodology, he said. "It's like flying when it's really foggy and you tend to rely on instruments rather than looking out the window. It's great, unless the radar is on the fritz. Social scientists are captivated by the radar screen. They don't look out the window anymore."
Incompetence, Confusion Can Lead One Astray
Most bad statistics are less the result of deliberate deception than incompetence, confusion or self-serving selection of numbers that reaffirm one's beliefs, said Joel Best, professor of sociology and criminal justice at the University of Delaware and author of "Damned Lies and Statistics" (University of California Press, 2001).
Best cites the example of a mathematically illiterate PhD student who mangled a figure found in a journal and wrote that "Every year since 1950, the number of American children gunned down has doubled," a figure that would have surpassed 35 trillion if the student had calculated the implication of the wording.
"A definition is everything," he said. "Some studies have found 90% of workers are bullied in the workplace. Others show 90% of students are bullied in schools. If you define bully broadly enough as people being mean to you, it's amazing 10% of the population has not been bullied."
A classic problem leading to distorted numbers arises when scientists report a cause-and-effect relationship between two factors but fail to account for other factors that could produce the same result. For instance, some people concluded from large-scale studies associating church attendance with low levels of mortality that doctors should prescribe religious activities. However, critics said the relationship could just as easily have resulted from the fact that people who were already seriously ill would be more likely to stay home.
Though not all readers of "startling new information" from a just-released study might have the savvy to question the study's basic assumptions, fellow researchers in a field are well positioned to. Recently, however, scientists--under increasing pressure to produce newsworthy results--have started marketing their results to the general public, bypassing the traditional safeguards provided by peer reviewed journals. "That's how you make your own work more visible. That's how you get that next grant," Best said.
Media, tempted by large, surprising or reasonable-sounding figures, sometimes go along repeating numbers without investigating--or mentioning--how they were collected, or whose agenda they might promote. When researchers come out with competing interpretations--such as colleagues involved in the now famous National Institute of Child Health and Human Development child-care study--journalists' leanings, Smith said, can lead them to find one more credible than another because it reinforces their beliefs or makes a more striking story.
Day-Care Study Was Reported as Bad News
Confusion may be one of the most enduring responses to the day-care study, which was released to the press last month during a conference of the Society for Research and Child Development. At first the study, the latest update in an ongoing look at the effects of day care on children, was reported as bad news. Presenting the results was researcher Jay Belsky, a day-care critic, who stressed the aggressive behavior found in 17% of kids who spent more than 30 hours a week in day care, compared with 6% who spent fewer than 10 hours a week away from their mothers. He suggested policies to extend parental leave and part-time work.
But Belsky's fellow researchers on this study rushed to point out the good news: The aggressive behavior the study reported was within the normal range for most children, and the study also showed language and memory benefits for the kids in high-quality care. They noted that 83% of the children are doing fine and that the study measures correlations, not causes. To prove cause and effect, social scientists would have to be able to randomly assign some parents and not others to use day care, for instance. And because they can't, their studies can only show links between two phenomena, not that one causesthe other.
The dust-up was variously blamed on public relations officials overeager for publicity; a hasty press conference by telephone; bad blood between Belsky and his working-mother colleagues; a desire to please the new, more conservative administration in the White House; and uncritical media interested only in quick sound bites.
Reached by telephone in London, where he is a psychology professor at the University of London's Birbeck College, Belsky said, "The truth is, most kids in whatever care are functioning in the normal range. We never stop to point that out."
Blaming colleagues who he said see modest findings as useless, he said, "All they want to say is good quality care is good for kids. I think that's true, but it's not the only thing that's true. When they show it, they don't qualify how modest the effect is, they don't point out how most kids in bad quality care function in the normal range, and they don't hesitate one iota in drawing policy implications and causal implications from correlational data."
The study's lead statistician, Margaret Burchinal, said she's never hidden her belief that children deserve better care. Part of scientific training, she said, is showing what you believe to be true in an unbiased way because it makes your case stronger.
Researchers said they had been advised to discuss only peer-reviewed reports in the future. As a result of the fuss, Belsky said, "I've come to believe that too much of social science research, especially as it gets disseminated, is ideology masquerading as science."
Reformers Historically Have Inflated Numbers
Some say he shouldn't be so surprised.
Historically, reformers have inflated numbers to help solve social problems, while public officials have tended to underestimate them, Best said. In 1833, New Yorkers concerned about prostitution, and pushing for authorities to take action, published a report stating there were 10,000 prostitutes in the city--a number equaling 10% of the female population. Other estimates ranged up to 50,000. In 1872, police counted 1,223 prostitutes.
Values, morality and religion fuel most debates about social issues, but people are more comfortable discussing numbers, researchers said. "Because of the supposed separation of church and state, we have a system that subscribes to an ideology of objectivity," said Judith Stacey, a USC sociology professor. "We're not supposed to argue from religion [or] from our personal values and impose it on others." Not so with "scientific research."
"Part of what's going on is we as a society have an appreciation for science, and there's the sense we ought to get the facts about one thing or another," Best said. Our society often mistakes statistics for facts, attributing magical powers to them, like fetishes, he said. Activists have learned to package their claims as facts by including numbers.
In many cases, he said, the number is "suspiciously round. Two million missing children. Three million homeless. Sometimes it's very difficult to figure out: How would you ever count these things? Of course the answer is, nobody is, and somebody is making a best guess. Usually if they think it's a big problem, they want a big number, and if they want to minimize it, they want a small number."
One commonly heard statistic is that gay teenagers are three times more likely to commit suicide than straight teenagers. However, Best said, coroners are not always sure which deaths are suicides and do not record sexual orientation on death certificates. "Nobody is keeping track of suicide statistics by groups who cannot be precisely identified themselves. How can anybody know? Maybe the rate is three times higher, maybe 17 times. Maybe half. There's no way to know that."
Over the past decade, as state and federal governments have begun enacting laws affecting family life, statistics about childbearing, child raising, marriage and divorce have become particularly contentious. "These days, almost any family research is rapidly politicized. It's scary to publish something," Stacey said.
For many years, for example, most research on gay and lesbian parenting was conducted by sympathetic researchers well aware that judges had been deciding against gay parents in child custody cases, Stacey said. The researchers "thought differences were deficits or that judges would seize on any differences, especially the possibility that boys will be sissies and girls will be tomboys. It's that popular prejudice that people were combating or avoiding," she said.
Perhaps unsurprisingly, their studies found no difference in children raised by straight parents and those raised by lesbian or gay parents. "I was suspicious of the no-difference claim," she said. "I found it very hard to believe that there wouldn't be a higher percentage who were less conforming gender-wise. I think that's a good thing."
The report she coauthored, published in April in the American Sociological Review, found that children raised by gay parents were more open to same-sex relationships, that boys were more sexually restrained and girls less so. The emotional health of the children was the same regardless of their parents' sexual orientation.
Still, a colleague, Steven Nock, professor of sociology at the University of Virginia, believes the samples in the studies she researched are too small to justify policy changes. Nock, who has been asked to testify in a Canadian case to determine marriage rights of gay and lesbian individuals, said, "We really haven't done the kind of research that would need to establish the presence or absence of any difference yet."
Research Used to Bolster Political Agendas
Many social scientists have been pushed out of academia into politics because of the implications of what Stacey calls "virtual social science"--spinning statistics to create "truth" in the mass media. In 1997, Stacey and several colleagues, worried about how judges and politicians were misusing research, helped found the Council on Contemporary Families, a group formed to educate the public on research about family diversity.
"You get into the political arena and there's no review," she said. "They just choose the research they want to have on the record. That's not research. That's politics."
In particular, what she called "fatherlessness dogma" appears in the preamble to the 1996 Personal Responsibility Act that reformed welfare: "The absence of a father in the life of a child has a negative effect on school performance and peer adjustment." Though there are studies to that effect, Stacey said, "you can find a study that says the opposite."
Currently, hearings to reauthorize the welfare reform bill are attracting pro-marriage "experts" she said. "That's not research," she said. "That's testimony."
Government funding has had a particular influence on researchers who depend on grants from Congress, researchers said.
Some government-funded researchers may hide or shade findings as "a truth that's just not useful at this point," said the Statistical Assessment Service's Murray. "You have to sniff the wind and find out which way this government is leaning and say the right things to keep the funds coming," he said.
In a particularly ironic twist, Scott O. Lilienfeld, an Emory University associate professor of psychology, said political pressure apparently affected a proposed journal article of his about political influence on child abuse research. His article was first accepted, then not accepted, then reaccepted by the American Psychological Assn.'s journal, American Psychologist.
Lilienfeld's article criticized the association for bowing to members of Congress--including Texas Republican Tom DeLay--who complained that a 1999 Psychological Bulletin article by Temple University Bruce Rind and others "normalized pedophilia." Rind's research analyzed existing studies of childhood sexual abuse and concluded that not all instances of sex between adults and children caused psychological harm to the children. In the wake of a media "feeding frenzy," the association apologized for the article and invited DeLay to contribute an article on child abuse.
Though Lilienfeld did not take a stand on Rind's findings, he said they did not condone pedophilia. The apology, he said, "raised the issues about potential suppression of unpopular or politically incendiary writings."
After media reports in journals Science and the Chronicle of Higher Education that Lilienfeld's article had been accepted in February and rejected in May, he received word Friday it would be published after all, in a special edition along with other commentaries.
Researchers Call for a Realistic Attitude
Some kind of "line in the sand" must be drawn between politics and social science research, Lilienfeld said. He and others argue that if researchers or scientific journals want to be political, it would be better to acknowledge their political stance rather than pretend they don't have one.
"If you acknowledge and reveal it, you subject yourself to a much higher level of scrutiny," Stacey said.
Because all numbers are a product of somebody and inherently flawed, the relationship between politics and research tends to support the postmodern philosophy that nothing can be known objectively anyway. The result, some say, is a loss of ideals.
But though the public may be tempted to throw up its hands, researchers call less for cynicism than for a more realistic and critical attitude about social science statistics. Though bad numbers are not helpful, to have no numbers, they said, is to have no guide at all to evaluating the scope or growth of society's problems.
The public needs to be particularly on guard when presented with statistics on politically charged topics, scrutinizing definitions of terms such as "intact family," or "sexual assault," researchers said. Consumers need to consider the source of a statistic and what it measures. No one should be particularly surprised, for instance, that gun control advocates find 75% of the population favors gun control, while surveys by the National Rifle Assn. find 75% support protecting the Second Amendment, Best said.
No single study will ever provide the last word on any social issue, researchers said. Said Lilienfeld: "Social science is a self-correcting process. The findings published may be right or wrong. That's the way science progresses."