What is hot or not used for?

Answer

Hot or Not is a website where users can rate the attractiveness of submitted photos. You can find matches in their ‘Meet Me’ engine and list your best pictures with extended features that give more information about you.

The Hot or Not Experiment

Hot or not is used to determine if something, based on a series of votes by individuals who like the thing or dislike it and find other people’s opinion, should be encouraged or discouraged. For example, we use hot and not voting in form of YouTube likes and dislikes when deciding what comments will be tolerated to save our users’ time from reading unnecessary comments. We also use hot or not for things such as promoting new videos posted by our users so that more people are exposed to their content.

What happened?

Last week I logged into my office computer and noticed I had received an email from Google. It was short but very alarming: I then read the blog post about how “Googlers must obtain approval from a project’s ethics committee” before implementing our hot or not feature on YouTube. I don’t know for sure why we had to do that exactly, but my guess is it’s because of the way YouTube implements the hot or not voting:

The cool thing about this implementation is that if you are working on a new idea (in Google they call them “moonshot” projects) and want to get quick feedback from your colleagues, you can add their email addresses as voters in the form above. Then whenever someone visits http://your-awesome-project/, it will ask their browser to send an email containing a link [the one above] with a unique ID number attached. Once each email is sent, the UI renders that ID next to your colleagues’ names. This way you can have a quick vote if an idea is cool before investing too much precious time in it. It’s also great for getting feedback from multiple people in a private and anonymous manner so that it doesn’t embarrass them when they don’t like your new project idea.

Why do I say our implementation on YouTube is different?

(Notice the “hot or not” title at the top.) On YouTube, we hide the person’s email (as well as the name), which allows us to show someone’s voting power, i.e., how many times they agreed with other people (or clicked something) instead of simply asking their opinion on something. In a nutshell, the feature is used to determine if people like or dislike a thing without knowing who exactly voted for what.

Is it illegal?

My understanding is that as long as you don’t use email addresses and mask them in your implementation (like we did on YouTube), then everything should be fine.

Why do I say “my understanding” Well because when I reported this issue to my manager was told

So, yep… he said it’s going to be alright so that’s what I’m sticking with :). But honestly Google never explicitly told us why we needed some sort of review from their ethics committee before implementing hot or not voting. So there is really no way to know if it’s actually legal or not.

Parah Dice – Hot (Official Music Video)

How did I find out about this in the first place?

It was right around our weekly “IP review” meeting, whereby coincidence another project manager reported a similar issue with their app:

So we were talking to each other and trying to figure out why this might be happening when one of us remembered reading a news article on how Google had just opened up an ethics review process for new projects:

But that is clearly speaking about new moonshot projects like Loon (I’m sure you’ve heard of that crazy idea to deliver internet access to remote areas via hot air balloons). It made me wonder if there would be some kind of problem with our hot or not voting implementation on YouTube. I later found out that the person who originally reported this issue to Google was working on a project called “Tide” (https://tide.withgoogle.com/) which, oh by the way, also uses [the exact same] hot or not implementation for getting feedback from their colleagues:

To make things even more curious, Tide is currently in private beta but will be released to the public soon. And interestingly enough, they have an ethics committee too! But before going any further let me show you what’s written there:

So it looks like Tide is actually releasing their product soon and not some kind of experiment or “moonshot” as we were led to believe…

Anyway, the important thing to note here is that Tide has all their voter’s email addresses and clearly uses them for sending out vote reminders, etc. So in my opinion, this is where Google’s problem started – not with YouTube. This is why I think it makes sense to give hot or not voting a chance (though of course, you should decide if it fits your project).

Whose job is it to go through each new feature before launching?

So probably it’s Google’s job but they are not the only ones who review projects at YouTube. In fact, lots of people do: PMs like me get feedback from dozens of co-workers daily when someone encounters a bug in one of our products or simply wants to suggest a new feature. Then we also have a dedicated “feature reviewer” team who does this:

(From left to right, as seen in that newsletter.)

So I guess what I’m saying is that most people are cool with hot or not voting (or actually kind of like the idea). And yet it appears only one person wanted me to get an ethics review before going live with such a feature… Why? Well because their manager told them so and said that there was some kind of ethical problem with hot or not voting on YouTube. So why don’t we try to figure out if there actually is something wrong with our setup or if someone simply doesn’t know how Google’s product release process works?!

I asked my manager for more details but he was unwilling to share any specifics. So I asked him if it was okay to ask someone from YouTube’s product marketing team about this issue with hot or not voting (since we had already talked to them earlier and told them a little bit about our implementation). He agreed so we got in touch with one of the “YouTube spokespersons” who reviewed all the votes on their site and confirmed that they were fine:

Check out the whole conversation here. You can see how surprised she is that Google’s project managers are telling engineers like me that they need an ethics committee pre-approval before implementing hot or not voting when clearly her team did those reviews without any problems.

Later, when we told the same thing to a “YouTube spokesperson” he was also surprised and put us in touch with someone who could help us figure out what was really going on here:

So it looks like Google’s internal culture has changed recently as they are now concerned about doing anything which might be considered “ethically controversial”. And while I’m not saying that censorship is good, that doesn’t mean we should stop engineers from trying new things. Especially when those engineers just want to make their users happy! But this guy makes some good points so let me tell you why our implementation is different and why I think it’s okay for YouTube and other companies to implement hot or not voting into their products:

We don’t allow users to vote based on appearance. So you can only vote if you watched the whole video. You can see the voting UI here:

(Click on image to enlarge.) We have a non-profit organization in charge of keeping our community’s voting behavior ethical. This might not be important for websites such as YouTube, Twitter, or Facebook but it’s crucial for us since we make money from donations and want to stay honest. In addition, we use an anti-cheating mechanism where suspicious votes are detected and automatically removed. (You can read more about this in this article: “Hot Or Not – Fighting Cheaters”.) My project is just a small experiment that should help me figure out how much my users like the idea of hot or not voting. And if it works well then we can improve it and eventually launch a full-scale implementation which will be great for everyone. My project violates none of Google’s policies since no one was fired based on their appearance. And even if someone was fired, they would have been fired anyway because of something in their work performance — i.e., not because of how they look!

So I have no illusions that this post will make things better and thus save my project right now (as we’re already past the deadline). But maybe next time someone wants to build a new feature that helps users vote on each other, there won’t be problems?

For example, this tool looks very promising and could be great for a lot of websites. So why not implement it at some point? UPDATE: Some people are asking me if I could implement the same voting system without using hot or not’s API and my answer is “yes, we can always do that” but it requires more development time as we would have to build everything from scratch and use our own database. So I’m just saying that you shouldn’t remove features from YouTube or other Google products because someone got offended (even if they have a point). Instead, improve how those features work, and then everyone will be happy!

FYI: If you want me to talk about this subject on Quora then send your questions here. And if you like this post then make sure you share it with your friends on Facebook, Twitter, or Google+! Also, if you have any questions about hot or not voting then please put them in the comments below.