Sunday, February 7, 2016

Bad Assumptions Make Bad Science

I was reading through on line articles from Scientific American recently and happened to stumble across three essays in a row that demonstrated precisely what is wrong with much of popular science. Well, two things, but I suppose "science as click bait" is a topic for another essay, so let us focus on the more interesting topic, the way that poorly designed experiments can easily produce misleading results, or, in some cases, even lead exclusively to the desired outcome. In other words, carefully mis-designing an experiment can produce whatever outcomes one desires*.

The first essay I saw claimed that mean were attracted to nonconformist women. That sounds reasonable, until one examines the study. Apparently, men were presented with a large number of online profiles and asked to select women. Out of this, they chose women the researchers defined ass "nonconformist" more often than expected. Thus, the conclusion was, though conventional wisdom says men would be attracted to conformist women, they are not**.

Hold on a moment! First, the study actually measured little but he first choice for a possible date. It says nothing about actual attraction or long term prospects. These women caught the eye of someone, there is no guarantee these men will continue dating them. So, while being different may give a momentary advantage, nothing says the nonconformists won't fare poorly in going beyond that first call. Thus, the study is much more limited than it appears.

Second, the study seems designed to produce this result. After all, if you were presented with, say, a mug book of 500 people, and all but one was white, who would you recall? The one non-white face! It is just human nature that, when presented with too many things to hold in our mind at the same time, we recall the anomalies, the oddities. And so, if the study -- as it should -- did not ask for the selection until everyone was viewed, then it seems of necessity the oddballs would be the ones the men would recall. Thus, it seems the design would skew toward this outcome.

Lastly, the experiment, for that matter any experiment predicated upon nebulous terms such as "nonconformist", leaves a lot of room for bias on the part of researchers. If not defined until after the selection, then obviously it is tempting to find a reason to call anyone selected a nonconformist. If the better design is chosen, with nonconformists so designated from the start, there is a temptation to pick the most attractive nonconformists and plain conformists, or, barring that, to define the most obviously attractive figures as nonconformist. In short, because the term is so nebulous, and there is usually some common ground in what people consider attractive, to some degree, it is easy to skew such experiments either consciously or unconsciously. Making me doubt any such outcomes.

The second essay had much more obvious problems, one of which was even pointed out in the comments. This essay was about studies finding that mean and women have differing ideas concerning whether they can be "just friends", and finding, by and large, men are horny beasts without a soul, and women are pure angels... no, wait, men are stalkers and women lunkheaded pollyannas?... No, I mean, men tend to have romantic feelings toward their female friends, and overestimate the attraction their friends feel toward them, while women are oblivious to this entire situation.

The first problem is perhaps the most obvious  -- well, if we ignore the ludicrously small sample size, but that seems to happen a lot in psychological studies, so I always just assume if there are more than 2 subjects, psychology considers it valid -- is the choice of a pool of subjects. The study was conducted among pairs of friends of the opposite sex -- so far so good -- among college undergraduates....

Hold the phone! So, people who think naked beer bongs at Daytona Beach is the height of culture, who think romance is taking your pants all the way off, and who consider fidelity letting her get home before your next conquest, or maybe not sleeping with her roommate, at least not while she's home -- they are to be exemplars for all of humankind? Yes, I think I see a little problem here...

Besides choosing test subject that make rabbits say "I think you're a little too fixated on sex", there is another problem. The test relies entirely on self reporting. In other words, they simply asked these subjects how they feel about their friend. Oh, the essay told how they made sure the friends would not talk about it, and so on, but let's face it, people lie. Even when they think it will never get out, even when they promise not to, even when they are getting paid for the truth. Just imagine asking a stranger if he ever touched another man's penis. Now, in life, this has probably happened to men in any number of relatively innocent contexts, moving too quickly in a locker room, bumping someone in a pool (we did not specify it had to be unclothed), for that matter, being jammed into a crowded subway train... But if you ask someone, unless he is prone to such activity, he is likely going to automatically deny it. And there are any number of such topics, and for many women, sexual feelings for a guys with whom they are "just friends" (or in another study, trying to poach their friend's boyfriend***) is such a subject.

One female comment writer pointed out something quite similar. As she said, women are taught throughout life, they must not give the impression they are "easy", and so will often deny they are interested in a given man, or most men. And that is the problem here. Even if they feel the results will remain confidential, there are women who will still not answer honestly. And men as well, the tendency to over-sexualize our view of women when speaking to other men, may have skewed those results as well. In short, there is simply no way to take these results seriously, as there are simply too many reasons why the self reporting could be deceptive. And if the self-reporting is unreliable, with no way to check it against something else, then the study as a whole is worthless.

The final essay, seemingly the least problematic, had the biggest, but best concealed, issues. This study claimed to take examples form three sports (baseball, basketball and soccer) and, using various performance statistics, argued that the teams with the most "high talent" players actually performed worse than would be predicted. In short, high talent players imposed costs and teams did better with more mediocre players.

Initially, upon reading this, I drew an egalitarian/socialist conclusion from this, arguing it is covertly political, making the case that just because one is better at something, he does not deserve better pay, as he may not be a benefit after all. Then I thought, perhaps I was grasping at straws, maybe it was just a study, and the obvious (to me anyway) motive was not driving the people who designed it. At least until I found this comment:

And, reading that, I decided I had to concede the point. It was not just me, the study struck at least one other -- way on the other end of the spectrum -- the same way.

Regardless of motive, there are quite a few good example, even from the world of sports, to offer as counter evidence -- the 1927 Yankess (Ruth, Gherig, Meusel, Lazzzeri, Combs, etc.) come to mind -- and it gets worse if we move to other areas. For example, the Second World War -- the German Generals in the 1939-1940 era were, arguably, the most talented available -- Rommel, Guderian, etc. -- and had terrific results. And then, in 1944-1945, the US and Russian armies both fielded some quite talented men, with equally great success. In fact, despite many examples of the personality clashes that supposedly make star sports figures so costly -- Patton, Ike, Bradley and company still managed to cooperate relatively well and wage a successful war. All of which seems to contradict this argument.

But that points me to the first problem of this study. How well does athletics mirror human behavior in any other areas? It is obvious why many scientists would try to use sports as an analogy. Sports, unlike most of life, provides us with an arena with fixed, known rules, clear winning and losing conditions, and a host of handy statistics. But, it is also a very artificial environment. One quite unlike life. In life, if one is poor at teamwork, he can always choose a field where individual initiative is useful, that is not possible in a team sport. Similarly, team sports usually have players largely of the same age range, with only a decade, at most two, difference in experience. In life, that is not very common, and experience plays a much larger role. Players also have a largely flat hierarchy, with a coach above them all, while in the real world, there are many more layers of authority, which makes for a quite different situation for those with differing degrees of talent. I could go on, but the point is simple, sports may make a good tool for analyzing some specific situations, but as a tool for analyzing, say, business, it seems to fall short.

Actually, using sports does present one other problem of this study, the way the artificial environment of sports skews things. Many sports, for example, have salary caps. Since players of high talent cost more, this means those teams with a few "superstars" have to skimp on supporting players. Even without salary caps, the economics of the game still make the buying of star players likely to force economizing elsewhere. Thus, a sports team with tops stars will probably have holes in its roster. This is a result of the artificial nature of sports. Businesses, governments, armies and the like are not bound by this limitation, and thus could have both a number of talented individuals and a good set of supporting players. So one part of the results seems to be an outcome determined by the study setup itself.

Which leads to the next complaint, one raised in the comments. The measurement used tended to favor those who scored the most, or achieved other individual accomplishments, and ignored assists and other cooperative acts which still show talent. In short, it was skewed to ignore any who used their talents to assist others. In other words, it limited the definition of talent to a subset of truly skilled players, and thus, tended to hide the true amount of talent on teams. In fact, some argued, by choosing the metric they did, they pretty much created their outcome.

In each study there were other issues, but to go into every one would make for a long, dull essay. Let us instead stop here and ask ourselves, why on earth would a reputable (for the most part) publication like SA publish such rubbish research? And, sadly, I think it is because, for the most part, scientists do not take the humanities seriously. And with good reason. These studies, sadly, are not unusual examples of bad methodology, sad outliers showing uncommon mistakes. These are the meat and potatoes of social sciences research. Such silly, inconsequential, and poorly designed studies are, sadly, the bulk of social science research, and for the most part, they are not only produced by reputable schools, but also pass peer review and are published.

So, if you ever wondered why social sciences are such worthless subjects, why doctors in those fields hold such wrong headed beliefs and seem capable of justifying almost any theory, and cannot understand why conventional wisdom in academia seems to run contrary to both human experience and logic, you need look no farther. These studies, and their countless companion in the many journals of social science, they are the reason for so much of the nonsense we find in universities.


==========================================================

* A perfect example is to be found in confusing any correlation with causation, especially when one could easily posit confounding variables. For example, the infamous reports on how "pornography leads to rape" or "violent video games lead to violence", both of which (assuming the correlation even exists, of which I am not certain), forget that perhaps there is some personality trait which attracts individuals to both rape and porn, or violence in life and video games. In short, maybe the correlation has nothing to do with causation. (See "Correlation vs. Causation", "Correlation and Causation Revisted", "Violence and Culture", "Our Unique Age, A Tempting Falsehood" and "More Persistent Myths") Of course, these are hardly unique. For example, by using value of property damaged as a measure of storm strength, it is inevitable, as development increased and prices rose, storms would seem to grow continually worse. (See "The World's Most Stupid Bureaucrat") But it seems much of modern pop science is filled with errors of this sort. ("Certainty and Pop Science".)

** NOTE: After I wrote this, I realized the article I read had few details. I recall now I actually looked up a bit more on this topic, and so I am criticizing a few of these experiments as described in other sources I cannot currently locate. I will try tot rack them down later and add links.

*** It is funny this study would say men are more likely to be attracted to involved women, than women for involved men. There are countless stories of a fake wedding ring making men more attractive, not to mention endless comedy routines, suggesting the perception is not quite in tune with this study. For that matter, in a bit of fiction I wrote (as yet unpublished) I had a character saying, if you meet two women and are interested in one, the easiest way to get her attention is to flirt with her friend. Women, according to my character, seem to have some innate rivalry, at least at certain ages, and seeming to show interest in one, will tend to catch the attention of the other. Again, not proof, but it seems a lot of us are quite skeptical that women are less likely to try to attract attached men. I am well aware anecdote is not evidence (cf "The Plural of Anecdote is not Data"), but then again, we are putting anecdote against a small pool doing self reporting, so I think in this one case I may give anecdote the edge.


No comments:

Post a Comment