I worked in Integrity at Meta (then Facebook) between 2017 and 2021. During that time I used the XCheck system and fixed some of the ways it interacted with the Integrity product I was working on. For most of that time, there wasn’t a dedicated team devoted to maintaining XCheck. This was normal for Meta at the time; there were more systems than there were teams and engineers to maintain them. Since I had written a few posts about it internally, I was often asked about XCheck internally. That changed around 2020 when a team started maintaining it.
In this post I’ll discuss why XCheck exists and what it does. This is accurate as of early 2021.
1. Integrity systems can be exploited
Imagine a system that allows users to report content. To start with users generally do a good job of reporting “bad” content, for their own specific definitions of bad. The number of reports number in the millions per day. That’s too many for human moderators so some element of automation needs to be introduced. Maybe after 5 or 10 or 100 reports, the system will automatically make the reported content invisible.
That works well for about two weeks until users figure out how to exploit this. They form groups that agree to coordinate to report content. Now reports become lower signal than before, but still somewhat useful. We use the reports, but try to limit exploitation.
2. Integrity Systems are not built for outliers
Any behaviour that looks like an outlier will also be displayed by legitimate users. For example, scammers rapidly send messages to people they’re not friends with a low rate of response. Therefore we might consider this behaviour suspicious and penalise such users. But from personal experience I can tell you that there are legitimate users of Facebook that do exactly this. They end up being penalised because their behaviour looks scammer-like.
This is why no system I’ve ever seen works with 100% precision and recall. Often, a system that exists and mostly works is preferred to a hypothetical perfect system that is never built. Here’s an example - the system that prevents unauthorised logins stops allowing any logins into an account after a certain number of failed attempts. This affects celebrities more than regular folks. That’s why Mark Zuckerberg cannot log into his account on a new device - it looks like just another account compromise attempt. As far as I know, there’s no plan to create a system that allows him (and other celebs) to log in easily. Time is better spent on preventing account compromise for regular folks.
When it comes to fake reports, celebrities get more of those than regular people. Sports fans reporting players they don’t like, or people reporting politicians they don’t like - that’s fairly common.
3. Integrity systems don’t always compose well
From the outside it might seem like Integrity within Meta is a few teams of maybe a couple hundred people sitting next to each other but that’s not the case. It’s thousands of people in hundreds of teams across multiple time zones. In 4 years I met and spoke to only a tiny fraction of them. Coordinating everyone is difficult.
If each of these teams built 4-5 components that were 99% accurate, the system as a whole probably end up penalising millions of genuine users.
Poor coordination can also be exploited. I’ll give a real example from 2019. At some point some team managing ID verification added a tag to user accounts saying Shielded-ID-Verified if the user’s ID had been verified by a human. Separately, some other team made the decision to exempt some accounts from the fake account checkpoint if there were good signals that they were real. ID verification seemed like a good signal, not controlled by a hypothetical abusive user.
Except, it could be. What some users figured out was that they could report their own fake account for impersonation. Their account would then be asked to prove their identity. They would pass ID verification by supplying a real ID. Now their account had the Shielded-ID-Verified tag and was exempt from the fake account checkpoint. This account and the thousands of others like it were free to abuse the platform. I fixed this loophole, but I’m sure others like it exist. It’s inevitable when there’s a lack of coordination.
Why XCheck exists
If we know the system can be exploited and it doesn’t handle outliers well, it makes sense to build a failsafe. A system to say “this user is good, we verified it, so don’t penalise this user”. That way the users who are most likely to receive fake/motivated/coordinated reports are protected to some degree. They’re also protected from bugs in integrity systems.
That’s what XCheck is. It’s a system to prevent accidental integrity enforcement from affecting accounts that are known to be good. This system is distinct from the blue checks you may see. Accounts are tagged with specific tags that exempt them from specific integrity enforcement.
This can and does go wrong. Shielded-ID-Verified was an XCheck tag. That tag is deprecated and not used any more, but XCheck remains. Another example in 2019, the footballer Neymar posted nude photos to Instagram. Normally such content would be taken down after sufficient number of reports were received. But his profile was protected by an XCheck tag that stopped the takedown process from completing successfully. The content had to be taken down manually by an employee overriding XCheck.
The response to these exploits is usually to fix that particular issue, like deprecating a tag that had been abused, rather than removing the system as a whole.
XCheck gets a bad rap (even internally) because we only hear about it when things go wrong. You never hear about the millions of instances where it worked well, preventing accidental enforcement on good accounts.
Why XCheck is in the news
The fact there is a two-tier enforcement system is newsworthy. Why am I, average person, treated differently from Neymar? It is hard to make a one-size-fits-all system, but it still feels wrong when you find out that it exists.
- Facebook Says Its Rules Apply to All. Company Documents Reveal a Secret Elite That’s Exempt. - Wall Street Journal, 2021.
- If BJP’s Amit Malviya Reports Your Post, Instagram Will Take it Down – No Questions Asked - The Wire, 2022.
The Wire’s report points out that the politician in question has a much smaller following than Neymar, and yet benefits from XCheck. But following is not the only criteria for applying this system. All of the Wire’s journalists and the Wire’s own account are protected by XCheck. None of their content will get taken down no matter how many haters report them. That’s working as intended.
XCheck is also not all-or-nothing. What matters is the specific tag and which enforcement the tag makes an account exempt from.
As of June 2021 no XCheck tag I knew of allowed a user to take down content at will. Other current and former employees (Alex Stamos, Sophie Zhang) have analysed the evidence and concluded it’s probably been faked. I agree with their assessment. More than any specific issue, I found the whole thing implausible. Instead of building a specific endpoint to share files or using a simple solution like Google Docs, Meta allegedly hacked around their SSO to create a separate workplace instance (instagram.workplace.com). This instance is so secret, no employee seems to have heard of it. And yet, this is apparently the primary mechanism of sharing information with law enforcement. Like I said, implausible.
Conclusion
- The platform is open to abuse. Integrity systems try to fix that.
- Integrity systems are open to abuse. XCheck tries to fix that.
- XCheck is open to abuse. And each instance of abuse is fixed when journalists report it. That’s why journalists should continue to write about it.
That said, if the story sounds too good to be true for the journalist, it probably is. If you really, really want the story to be true, it’s probably not.