I’m the first to admit it: we might be popular, we might create a lot of great relationships, we might blah blah blah. But OkCupid doesn’t really know what it’s doing. Neither does any other website. It’s not like people have been building these things for very long, or you can go look up a blueprint or something. Most ideas are bad. Even good ideas could be better. Experiments are how you sort all this out. Like this young buck, trying to get a potato to cry.
We noticed recently that people didn’t like it when Facebook “experimented” with their news feed. Even the FTC is getting involved. But guess what, everybody: if you use the Internet, you’re the subject of hundreds of experiments at any given time, on every site. That’s how websites work.
Here are a few of the more interesting experiments OkCupid has run.
Experiment 1: LOVE IS BLIND, OR SHOULD BE
OkCupid’s ten-year history has been the epitome of the old saying: two steps forward, one total fiasco. A while ago, we had the genius idea of an app that set up blind dates; we spent a year and a half on it, and it was gone from the app store in six months.
Of course, being geniuses, we chose to celebrate the app’s release by removing all the pictures from OkCupid on launch day. “Love Is Blind Day” on OkCupid—January 15, 2013.
All our site metrics were way down during the “celebration”, for example:
But by comparing Love Is Blind Day to a normal Tuesday, we learned some very interesting things. In those 7 hours without photos:
And it wasn’t that “looks weren’t important” to the users who’d chosen to stick around. When the photos were restored at 4PM, 2,200 people were in the middle of conversations that had started “blind”. Those conversations melted away. The goodness was gone, in fact worse than gone. It was like we’d turned on the bright lights at the bar at midnight.
This whole episode made me curious, so I went and looked up the data for the people who had actually used the blind date app. I found a similar thing: once they got to the date, they had a good time more or less regardless of how good-looking their partner was. Here’s the female side of the experience (the male is very similar).
Oddly, it appears that having a better-looking blind date made women slightly less happy—my operating theory is that hotter guys were assholes more often. Anyhow, the fascinating thing is the online reaction of those exact same women was just as judgmental as everyone else’s:
Basically, people are exactly as shallow as their technology allows them to be.
Experiment 2: SO WHAT’S A PICTURE WORTH?
All dating sites let users rate profiles, and OkCupid’s original system gave people two separate scales for judging each other, “personality” and “looks.”
I found this old screenshot. The “loading” icon over the picture pretty much sums up our first four years. Anyhow, here’s the vote system:
Our thinking was that a person might not be classically gorgeous or handsome but could still be cool, and we wanted to recognize that, which just goes to show that when OkCupid started out, the only thing with more bugs than our HTML was our understanding of human nature.
Here’s some data I dug up from the backup tapes. Each dot here is a person. The two scores are within a half point of each other for 92% of the sample after just 25 votes (and that percentage approaches 100% as vote totals get higher).
In short, according to our users, “looks” and “personality” were the same thing, which of course makes perfect sense because, you know, this young female account holder, with a 99th percentile personality:
…and whose profile, by the way, contained no text, is just so obviously a really cool person to hang out and talk to and clutch driftwood with.
After we got rid of the two scales, and replaced it with just one, we ran a direct experiment to confirm our hunch—that people just look at the picture. We took a small sample of users and half the time we showed them, we hid their profile text. That generated two independent sets of scores for each profile, one score for “the picture and the text together” and one for “the picture alone.” Here’s how they compare. Again, each dot is a user. Essentially, the text is less than 10% of what people think of you.
So, your picture is worth that fabled thousand words, but your actual words are worth…almost nothing.
Experiment 3: THE POWER OF SUGGESTION
The ultimate question at OkCupid is, does this thing even work? By all our internal measures, the “match percentage” we calculate for users is very good at predicting relationships. It correlates with message success, conversation length, whether people actually exchange contact information, and so on. But in the back of our minds, there’s always been the possibility: maybe it works just because we tell people it does. Maybe people just like each other because they think they’re supposed to? Like how Jay-Z still sells albums?
To test this, we took pairs of bad matches (actual 30% match) and told them they were exceptionally good for each other (displaying a 90% match.)† Not surprisingly, the users sent more first messages when we said they were compatible. After all, that’s what the site teaches you to do.
But we took the analysis one step deeper. We asked: does the displayed match percentage cause more than just that first message—does the mere suggestion cause people to actually like each other? As far as we can measure, yes, it does.
When we tell people they are a good match, they act as if they are. Even when they should be wrong for each other.
The four-message threshold is our internal measure for a real conversation. And though the data is noisier, this same “higher display means more success” pattern seems to hold when you look at contact information exchanges, too.
This got us worried—maybe our matching algorithm was just garbage and it’s only the power of suggestion that brings people together. So we tested things the other way, too: we told people who were actually good for each other, that they were bad, and watched what happened.
Here’s the whole scope of results (I’m using the odds of exchanging four messages number here):
As you can see, the ideal situation is the lower right: to both be told you’re a good match, and at the same time actually be one. OkCupid definitely works, but that’s not the whole story. And if you have to choose only one or the other, the mere myth of compatibility works just as well as the truth. Thus the career of someone like Doctor Oz, in a nutshell. And, of course, to some degree, mine.
![](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="500" height="806"><rect fill-opacity="0"/></svg>)
|
If you like the this blog, you'll love this book: Dataclysm. I spent the last three years writing it. You should buy it here! |
I weep for humanity.
This was a really fascinating bit of analysis, but it is kinda depressing. I am sure many people will go ‘but I really do care about the text!’, but it is kinda besides the point, and chances are, they do not nearly as much as their self image says they do.
I wonder if this means online dating is, long term, doomed.. or if there is some technique or pattern out there that will work better then meatspace.
Too funny! Just found out about this blog yesterday and was so disappointed that it had been defunct for the last 3 years. Glad to see it’s back. Will be reading frequently!
So cool you’re back.
Great article. As for the above comment, I can say that in my experience (straight male) perfecting the art of writing a good profile has made a marked difference. Having said that, getting my pics right has made a big difference to (thank you MyBestFace, and thank you puppy in Vietnam!).
J.
I know from my perspective that I see a definite difference between what you talk about doing here, and what facebook did.
Facebook secretly manipulated the users of the site, attempting to alter their emotions. Quite possibly at the behest of creepy interests sponsoring this experiment.
All I can see here is honest curiosity and a desire to improve the site. Completely different things. When you start trying to match people who are wrong with each other, just to see if you can play god, then I’ll have a problem with it.
Thanks for bringing this blog back.
I love this kind of analysis, and it addresses the kinds of questions that anyone who has used this site has had on occasion.
I imagine that all the dating sites have probably done similar analyses, but they never share them with the users because they’re “trade secrets” and, probably (and more importantly) they reveal some of the shortcomings of the other (mostly for-pay) sites.
Keep it up, guys…
Awesome stuff! Great you brought the blog back! Please keep it up (or send me a DB dump, I’d just dig around it on my own)
I guess I’m the exception here.
When I was reading over the “people pay more attention to pictures” bit, the whole time I was thinking of all the text-less profiles I’ve ignored (regardless of match percentage or looks or whether she lived nearby) and how 90%+ of the first messages I’ve sent were because something in the *text* stood out to me as saying “hey, this person might be pretty great” (…and on the other hand, how there’s been plenty of others whose looks I liked and whose answers to questions didn’t have anything super unacceptable, but I just couldn’t think of anything to say because the profile text just… wasn’t that special, leading me to believe that their personalities might be just as boring!)
I want to be surprised by these results, but I have to admit that I honestly am not. Outside of the organic way of making acquaintances through proximity x time x some kind of common interest, people are going to have a strong tendency to default to first impression, and if they are not particularly happy with what they see they will move on. “Plenty of fish in the sea” is a very easy mentality to acquire in an online world with a seemingly endless number of matches and the “perfect” person is surely just a few profiles away.
From a CogSci major with deep seated interests in sociology and psychology: are you guys hiring?
Cool that you guys are giving this some life again. As a Computer Science Master, I thoroughly enjoyed reading the previous entries that were posted. It really gives a unique insight into how people behave and react (and especially tips and tricks ;p).
Haha, I would love to get my hand on the datasets you use, just to poke around and see if there are any interesting patterns to discover.
Hope too see more soon!
I hope you didn’t cause me to miss out on a relationship while playing around with the data. It is hard enough for me as it is. Also , people trusting your match percent is what you want, so don’t lie to them about it.
Good to know at any one time here forward I can trust or distrust your Match %s, not knowing if your lab experiment has resurfaced.
How big were the sample sizes for these? It seems like it would make a big difference to the significance of the findings…
I picked an outrageous username, and then made my profile text very different from my username. Guess what? over half the people I emailed didn’t even look at my profile before responding. They would only respond with “your username is ridiculous”
It’s probably not a good idea to experiment on live human subjects, tho. That’s unethical.
However, if you’re only experimenting on hot women with no profile text (BOTs!), then I say, GO FOR IT.
Awesome it’s back! Now step it (back) up and get us those neat interactive stats from way back when for next time
I’d like to see the actual female : male ratio on any dating site.
I feel that the reason we are more “popular” than guys is simply due to dating site has way more male demographics than female.
The “look is almost everything” result really isn’t too surprising. We are human after all. Most of us are here due to no small part to carnal desires.
When’s the Kindle version of the book coming out?
@SteveRestless
What if the manipulation is used for what you believe does good for the person. That is what I am forced to believe is the reason, people end up manipulating people for money or good in their own heart.
I think people are naive when looking at online dating sites. Most of the time the people that really use it are just looking for someone to have sex with. The people that actually want to find their partner in life, are out numbered by the other population and thus the stats show it.
So.. I’m paying you so you can fuck with me? I am talking about the last experiment. Yes, I tend to message women who show a very high “match” percentage with me and I have never heard back from anyone. On the other hand, I have met some really cool women who you said “Ya’ll have issues”…. Now that makes sense.
Awesome to see a new post!
I started online dating in the summer of 2000–I’m not sure if the site is even around anymore. After three years or so of red-hot success, I got in an 8-year relationship with someone I met online. It ended early this year so I’ve jumped back in: and things are VERY different.![:)](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="15" height="15"><rect fill-opacity="0"/></svg>)
The point of online dating is not to meet people; it’s to meet weird people. By that I mean, people who also are obsessed with Iranian cinema or Hummel figurines or lake biology or whatever. You should use online to look for things that are NOT apparent at a bar/general meetup/other offline activity. If you go by looks/don’t read the profile, especially if you’re a guy, you’re stupid. Go to a bar and get laid a lot more often.
Also, OKC, by enabling mobile use you’ve destroyed a lot of value of the site. It is much harder to exchange meaningful messages or build rapport before meeting. Furthermore, you’ve made it fairly difficult to find casual sex partners on the site, which is the other giant thing you use online for (again, because it’s not something that’s easy to ask at a bar). So it’s kind of the worst of two worlds.
Here’s how I’d innovate. Let users choose a version without pics. However, they *also* upload an anonymous pic associated invisibly with their account, which other users can rate BY ITSELF without knowing the profile it belongs to. The results are displayed to users searching the pic-less profiles (“Most users considered this person to be Very Attractive” or whatever). Then let the profiles speak for themselves. I think the value-add here, and for online dating in general, is to steer people toward making better relationship choices than they otherwise would. That starts with emphasizing things like the weirdness you each have in common
Why don’t you admit to using bots & what is the point of annoying people with bots that are using your site? It’s hard enough to actually meet someone here, they mostly seem stuck up & not really interested in any relationship, they simply want to advertise themselves, mostly untruthfully, for their own satisfaction. I’m not getting it….
Hey Christian, I just got married to a man I met on OKCupid in 2010. I tried to log back in and retrieve our tests/match information because we thought that would be “cute” information to have or put in our insufferable scrapbook or whatever. Sadly, the accounts have been superdeleted (obviously) (if one of ours hadn’t been, that’s probably a problem). Is there a way I can importune someone who works with you to get us our match info? WAS SCIENCE RIGHT OR WRONG?
Love,
Beth and Justin (twowordheadline/nordicglow and holypopejustin)
So the key to improving your engagement metrics is to overestimate everyone’s match percentage![:)](data:image/svg+xml,<svg xmlns="http://www.w3.org/2000/svg" width="15" height="15"><rect fill-opacity="0"/></svg>)
I only read the site for the articles..
Glad to see the analysis posts back! How can this account for people who get a lot of views, and few/no messages, or mutual matches and then no messages? I assumption it means they rate you highly based on the pictures, but then when they actually read the profile they see something they don’t like. This study says the text doesn’t matter, so then what would explain the low conversion rate between views and messages? I agree with the person who said that the mobile version has degraded the quality of the site. I think between that and Quickmatch there are a lot of casual users who just scroll through the pictures but are not interested in actually meeting someone.