With election interference investigations going full force in the US, social media companies are being pushed to find new, fast ways to identify government-linked bots and trolls on their platforms — Russian ones in particular.
On April 10, Reddit announced plans to publish a list of accounts that the company suspected of having links to Russia's Internet Research Agency, commonly referred to as a “troll factory”. Reddit's in-house “Trust & Safety” and “Anti-Evil” teams identified these accounts in cooperation with broader inquiries by the US Congress into electoral interference on social media.
Multiple users were then banned from the platform. But not all of these users had provable links to the Kremlin.
On April 12, one user who lost their account implored Reddit to check its work:
oi @reddit check your admin's inbox on the website. your recent list of 944 suspicious users has 3 of my accounts and none of them are “Russian bots” or “Russian trolls”. I just happen to be from Russia. UNBAN ME.
— ☆ (@ajcrwl) 12 апреля 2018 г.
Reddit's quick-fix approach to bot-hunting seemed to have dragged a number of innocent victims into its nets. For this user, it appears that the key “suspicious” thing about their account was their location — Russia.
What made this possible? Using custom Python scripts and open-source research tools, I decided to take a closer look at these accounts, in an effort to understand what led to this outcome.
At the outset, I could see that the distribution of their creation dates was typical of a troll factory's intensive output: an overall period of registrations ran from May 2015 to October 2016, with large peaks of activity from May to July 2015.
This was an unsurprising finding, and a possible reason that the accounts might have triggered the system.
Next, I wanted to see whether the Reddit usernames had been re-used on other social media sites. I ran a script to query both Twitter and the Russian social network VKontakte (VK) for matching accounts. Although VK didn't return any of interest — only a couple that were likely coincidental hits — there were 106 common usernames live on Twitter.
On examining these Twitter accounts, two showed possible links pointing to the IRA troll farm of Savushkina 55 (in late 2017 their office moved to a different address in Saint Petersburg.)
First there was @shomyo, a sketchy profile with “john Doe” as a given name, following only six accounts, mostly Russian, including a Saint Petersburg football team. It was otherwise unused.
The second example pointed more substantively to the Internet Research Agency, but has been dormant since 2015. @rapitangnyy, with a linked pro-Kremlin LiveJournal account, tweeted links to similar patriotic blogs. Furthermore, this handle was included on a leaked list of IRA accounts from former IRA insider Lyudmila Savchuk released that year; it is quite possible that Reddit used this list to derive some of the indicators for their investigation. I published my own analysis of similar pro-Kremlin content on LiveJournal, also in that year.
The matching rapitangnyy Reddit account (archive) was largely used to promote the Russian-language LiveJournal blog posts, most of them painting Ukraine and the West in a bad light. It has been inactive for two years. So this is, without doubt, a Russian troll factory account — but not one deployed against the US during the election.
Perhaps the most noteworthy findings during my trawl through Reddit's “suspicious” users were those that likely weren't products of a troll factory at all.
Among the Russian accounts with a matching Twitter handle was @ajcrwl, a user giving their location as “Stuck in Omsk”. Their website was dedicated to digital art and graphic design.
I was intrigued — could this person be a remote employee of the infamous Saint Petersburg troll farm, industriously producing anti-American memes to order? Had their identity been revealed by their ill-judged reuse of the same username across multiple platforms? As previous work by RuNet Echo has shown, this would not be unprecedented.
After further research and verification, however, this prospect seemed less and less likely. Although the Reddit account /u/Ajcrwl was created in October 2016, during the broader troll activity period, the account itself showed no sign of any anti-US or pro-Kremlin activity; instead, there were posts concerning rose growing and the video game Dishonored 2.
ajcrwl, whom I noticed earlier tweeting at Reddit, was not acting like a guilty party. I contacted them a day later, curious to learn how they landed on Reddit's radar during the investigation.
“I don't think anyone from Reddit even bothered to check [my account(s)] outside of Reddit”, they responded.
When I asked whether they ever used a VPN, or interacted with the platform in an unusual way, they wrote:
Sometimes I use a VPN connection, and I think I've been flagged in January when I deactivated one of my Reddit accounts and immediately created another. There's a chance I logged into the third one back then as well. I've also been logged into two accounts from my phone.
ajcrwl willingly shared with me the AS number (a code that identifies a range of IP addresses used by a particular service provider) and VPN addresses (based in London, Poland and Atlanta) they had used before their ban. None of them matched with any of the infrastructure I have on file (my records, derived from open sources, are robust though not exhaustive) as being linked to the Internet Research Agency, RIA FAN, or the IP addresses of their associated forum trolls.
Rather, it seems more likely that ajcrwl's use of a VPN, in combination with activity involving three accounts, set off Reddit's tripwire.
On April 17, I returned to Reddit's suspicious accounts list and found that ajcrwl and three additional accounts had been removed from it, and unbanned. ajcrwl confirmed that two of the accounts also belonged to them.
The fourth account belonged to RobbyDelaware, an American based in the country of Georgia who was previously interviewed by VICE's Motherboard in November 2017 after his Twitter account, with an identical handle, was unexpectedly hit by the ban hammer.
Despite this publicity and apparent vindication, I noticed on May 9 that Delaware's Twitter account had been marked as restricted due to “unusual activity”.
On that same day, ajcrwl also found — to their chagrin — that their Reddit block appeared to be haunting them on Twitter:
Twitter has just locked me for "automated behavior" for no reason at all.
Unlocked easily, but I thought I should log this.
— ☆ (@ajcrwl) May 9, 2018
Through further analysis I discovered that Twitter had taken similar action against nearly all of the 106 accounts with usernames matching those on Reddit's “suspicious” list, even @LGBTUnited, a now-inactive LGBTQ-themed promoter with no apparent connection to the troll farm page of the same name. Some other accounts marked restricted at time of writing included apparently real people with no obvious ties to Russia, such as @hcaner, a growth hacker from Brazil.
Yet some of the accounts that appeared to be fake or spreading spam, such as @Meepopeep or @LaserAthletics, escaped this fate. Twitter appeared to be using the first revision of Reddit's “suspicious” list as its primary indicator, with little or no verification.
I got in touch with Robby Delaware, explaining that his account @robbydelaware — which he hadn't tweeted from since October 2017 — had been restricted.
In this exchange, I learned that Delware was one of the first Twitter users to draw attention to an early attempt by the Internet Research Agency to manipulate American audiences in 2014, actions which may ultimately have contributed to his blocks on both platforms.
When he noticed some unusual activity on the hashtag #ColumbianChemicals, he notified Twitter's security team:
.@twittersecurity There's a bogus #ColumbianChemicals tag (along with others) flooding twitter about fake chem. plant explosion.
— Robby Delaware (@RobbyDelaware) September 11, 2014
Bot and troll activity on #ColumbianChemicals was revealed the following year by the New York Times as an IRA-linked campaign. But Delaware had called it out right as it hit the web, even setting up a Pastebin of his sleuthing efforts (he also posts as iPad_App_Bugs).
This is me. I wrote down some of this stuff in a PasteBin post in 2014: https://t.co/J3H7hHPqv2 The goofy videos were from that day.
PasteBin post has around 100 Twitter accounts that were involved also.
— iPad Mini Bugs (@iPad_App_Bugs) March 2, 2018
Delaware has the rare distinction of being banned on Twitter then reinstated; banned on Reddit then reinstated; and then being censored once again on Twitter, apparently for attempting to draw attention to a pro-Kremlin campaign.
He received no explanation from either company, simply an acknowledgement from Reddit that they restored his account.
Unlike ajcrwl, Delaware told me he had never used a VPN, but a regular internet service provider in Georgia, his country of residence. It is possible that a Georgian IP, in combination with activity around a hashtag associated with a troll farm campaign, set off Twitter's algorithms — but a cursory manual inspection would made it clear why Delaware had used hashtag in the first place. He told me:
I do believe that Twitter used the most basic methods of automation to flag my account. I believe that the Columbian Chemicals hashtag, along with 3 tweets in Russian language, caused my account to be flagged.
Delaware in fact posted seven tweets in Russian from his original account — as indicated by ‘ru’ in the language field of his tweets’ metadata — but they were satirical or politically neutral, not genuinely pro-Kremlin. He also re-tweeted a tweet by pro-Kremlin Komsomolskaya Pravda journalist Dmitriy Smirnov showing a meeting between Vladimir Putin and former Former President of Kyrgyzstan Almazbek Atambayev. But context is vital — this alone is no indication that Delaware was sympathetic to his politics.
These two case studies suggest that while some counter-disinformation efforts by Reddit and Twitter appear to have been largely successful, their approach has brought significant collateral damage for a seemingly small group of regular users.
One can see many reasons why Reddit and Twitter would share such information, applying automation, or using the same sources to derive indicators of suspicious activity. But it is surprising to think that they might do so without critically interpreting this information with additional technical analysis, manual verification or open-source research. With such a relatively small number of accounts, individual perusal by human investigators would be feasible.
This strategy would lower the rate of incorrect bans and increase user confidence in the fight against disinformation, while reducing the risk of pro-Kremlin media capitalizing on these minor errors and presenting the investigations as baseless or illegitimate.