Which world leaders are twitter bots?
Published: March 28, 2018
Set-up
Given that I do quite like twitter, I thought it would be a good idea to right about R’s interface to the twitter API; {rtweet}. As usual, we can grab the package in the usual way. We’re also going to need the {tidyverse} for the analysis, {rvest} for some initial webscraping of twitter names, {lubridate} for some date manipulation and {stringr} for some minor text mining.
install.packages(c("rtweet", "tidyverse", "rvest", "lubridate"))
library("rtweet")
library("tidyverse")
library("rvest")
library("lubridate")
Getting the tweets
So, I could just write the names of twitter’s 10 most followed world leaders, but what would be the fun in that? We’re going to scrape them from twiplomacy using {rvest} and a chrome extension called selector gadget:
world_leaders = read_html("https://twiplomacy.com/ranking/the-50-most-followed-world-leaders-in-2017/")
lead_r = world_leaders %>%
html_nodes(".ranking-entry .ranking-user-name") %>%
html_text() %>%
str_replace_all("\\t|\\n|@", "")
head(lead_r)
## [1] "realdonaldtrump" "pontifex" "narendramodi" "pmoindia"
## [5] "potus" "whitehouse"
The string inside html_nodes() is gathered using selector gadget. See this great tutorial on {rvest} and for more on selector gadget read vignette("selectorgadget"). Tabs (\t) and linebreaks (\n) are removed with str_replace_all() from the {stringr} package.
Now we can collect the twitter data using {rtweet}. We can use the function lookup_users() to grab basic user info such as number of tweets, friends, favourites and followers. Obviously analysing all 50 leaders at once would be a pain. So we’re only going to take the top 10 (WARNING: this could take a while)
lead_r_info = lookup_users(lead_r[1:10])
lead_r_info
## # A tibble: 10 x 20
## user_id name screen_name location description url protected
## <chr> <chr> <chr> <chr> <chr> <chr> <lgl>
## 1 25073877 Donal… realDonaldT… Washing… 45th President o… http… FALSE
## 2 5007043… Pope … Pontifex Vatican… Welcome to the o… http… FALSE
## 3 18839785 Naren… narendramodi India Prime Minister o… http… FALSE
## 4 4717417… PMO I… PMOIndia "India " Office of the Pr… http… FALSE
## 5 8222156… Presi… POTUS Washing… 45th President o… http… FALSE
## 6 8222156… The W… WhiteHouse Washing… Welcome to @Whit… http… FALSE
## 7 68034431 Recep… RT_Erdogan Ankara,… Türkiye Cumhurba… <NA> FALSE
## 8 2196174… Sushm… SushmaSwaraj New Del… Minister of Exte… <NA> FALSE
## 9 3669871… Joko … jokowi Jakarta Akun resmi Joko … <NA> FALSE
## 10 44335525 HH Sh… HHShkMohd Dubai, … Official Tweets … http… FALSE
## # ... with 13 more variables: followers_count <int>, friends_count <int>,
## # listed_count <int>, statuses_count <int>, favourites_count <int>,
## # account_created_at <dttm>, verified <lgl>, profile_url <chr>,
## # profile_expanded_url <chr>, account_lang <chr>,
## # profile_banner_url <chr>, profile_background_url <chr>,
## # profile_image_url <chr>
We only want the columns of interest (name, followers_count, friends_count, statuses_count and favourites_count) and then we want the data in long format. To do this we’re going to use select() and gather()
(lead_r_info = lead_r_info %>%
select(name, followers_count, friends_count, statuses_count, favourites_count, screen_name) %>%
gather(type, value, -name, -screen_name))
## # A tibble: 40 x 4
## name screen_name type value
## <chr> <chr> <chr> <int>
## 1 Donald J. Trump realDonaldTrump followers_count 48426576
## 2 Pope Francis Pontifex followers_count 16858642
## 3 Narendra Modi narendramodi followers_count 40710139
## 4 PMO India PMOIndia followers_count 25156203
## 5 President Trump POTUS followers_count 22324998
## 6 The White House WhiteHouse followers_count 16713369
## 7 Recep Tayyip Erdoğan RT_Erdogan followers_count 12513987
## 8 Sushma Swaraj SushmaSwaraj followers_count 11461412
## 9 Joko Widodo jokowi followers_count 9693534
## 10 HH Sheikh Mohammed HHShkMohd followers_count 8764455
## # ... with 30 more rows
Now we can use the fantastic {ggplot} to plot the respective counts for each world leader
ggplot(data = lead_r_info,
aes(x = reorder(name, value), y = value,fill = type, colour = type)) +
geom_col() +
facet_wrap(~type, scales = "free") +
theme_minimal() +
theme(
strip.background = element_blank(),
strip.text = element_blank(),
title = element_blank(),
axis.text.x = element_blank()
) +
coord_flip() +
geom_text(aes(y = value, label = value), colour = "black", hjust = "inward")
Notice Donald trumps everyone in the followers and status area (from what I here he’s quite a prevalent tweeter), however Sushma Swaraj and Narendra Modi trump everyone when it comes to favourites and friends respectively.
Now, we’re going to use the function get_timelines() to retrieve the last 2000 tweets by each leader. Again this may take a while!
lead_r_tl = get_timelines(lead_r, n = 2000)
Unfortunately get_timelines() only gives us their twitter handle and doesn’t return their actual name. So I’m going to use select() and left_join() to add the column of names to make for easier reading on the upcoming graphs
names = select(lead_r_info, name, screen_name)
lead_r_twitt = left_join(lead_r_tl, names, by = "screen_name")
get_timelines() gives us the source of a persons tweet, i.e. iPhone, iPad, Android etc. So, what is the post popular tweet source among world leaders?
lead_r_twitt %>%
count(source) %>%
ggplot(aes(x = reorder(source, n), y = n)) +
geom_col(fill = "cornflowerblue") +
theme_minimal() +
theme(
strip.background = element_blank(),
axis.text.x = element_blank()
) +
labs(
x = NULL,
y = NULL,
title = "Tweet sources for world leaders"
) +
coord_flip() +
geom_text(aes(y = n, label = n), hjust = "inward")
Either world leaders really love iPhones or their social media / security teams do. Probably the latter. I can hear you all begging the question, using which source is more likely to give a world leader more retweets and favourites? To do this we’re going to summarise each source by it’s mean number of retweets and favourites and then gather the data into a long format for plotting
lead_r_twitt %>%
group_by(source) %>%
summarise(Retweet = mean(retweet_count), Favourite = mean(favorite_count)) %>%
gather(type,value,-source) %>%
ggplot(aes(x = reorder(source, value), y = value, fill = type)) +
geom_col() +
facet_wrap(~type, scales = "free") +
theme_minimal() +
labs(
x = "Source",
y = NULL,
title = "Which source is more likely to get more retweets and favourites?",
subtitle = "Values are the mean in each group"
) +
theme(
legend.position = "none",
axis.text.x = element_blank()
) +
geom_text(aes(y = value, label = round(value, 0)), colour = "black", hjust = "inward") +
coord_flip()
Naturally this leads me to the question of which leader, over their previous 2000 tweets, has the most overall retweets and favourites, and who has the highest average number of retweets and favourites?
lead_r_twitt %>%
group_by(name) %>%
summarise(rt_total = sum(retweet_count), fav_total = sum(favorite_count),
rt_mean = mean(retweet_count), fav_mean = mean(favorite_count)) %>%
gather(type, value, -name) %>%
ggplot(aes(x = reorder(name, value), y = value, fill = type)) +
geom_col() +
labs(
x = NULL,
y = NULL,
title= "Mean and total retweets/favourites for each world leader"
) +
coord_flip() +
facet_wrap(~type, scales = "free") +
theme_minimal()
What about the mean retweets and favourites per month? ts_plot() provides us with a quick way to turn the data into a time series plot. However this wouldn’t work for me so I’m doing it the dplyr way. I’m going to a monthly time series so first we need to aggregate our data into months. The function rollback(), from {lubridate}, is fantastic for this. It will roll a date back to the first day of that month whilst also getting rid of the time information.
lead_r_twit2 = lead_r_twitt %>%
mutate(year_month = rollback(created_at, roll_to_first = TRUE, preserve_hms = FALSE)) %>%
group_by(name, year_month) %>%
mutate(fav_mean = mean(favorite_count), rt_mean = mean(retweet_count))
We now have two columns, fav_mean and rt_mean, that have in them the mean number of retweets and favourites for each leader in each month. We can use select() and gather() to select the variables we want then turn this into long data for plotting
lead_r_twit2 = lead_r_twit2 %>%
select(name, year_month, fav_mean, rt_mean) %>%
gather(type, value, -name, -year_month)
Now we plot
lead_r_twit2 %>%
ggplot(aes(x = year_month, y = value, colour = name)) +
geom_line() +
facet_wrap(~type, scales = "free", nrow = 2) +
labs(x = NULL, y = NULL,
title = "Mean number of favourites/month for world leaders") +
theme_minimal()
Are world leaders actually bots?
{botrnot} is a fantastic package that uses machine learning to calculate the probability that a twitter user is a bot. So the obvious next question is, are our world leaders a bot or not?
We need to install the development package from GitHub and we also need to install the GitHub version of {rtweet}
devtools::install_github("mkearney/botrnot")
devtools::install_github("mkearney/rtweet")
library("botrnot")
library("rtweet")
The only function, botornot(), works on either given user names, or the output of the get_timelines() function from {rtweet}. To keep the inline with the rest of the blog, we’re going to use the output we’ve already created from get_timelines(), stored in lead_r_tl
bot = botornot(lead_r_tl) %>%
arrange(prob_bot)
For a clearer look at the probabilities I’m going to plot them with their actual names instead of the screen names
bot %>%
rename(screen_name = user) %>%
inner_join(distinct(names), by = "screen_name") %>%
select(name, prob_bot) %>%
arrange(prob_bot) %>%
ggplot() +
geom_col(aes(x = reorder(name, -prob_bot), y = prob_bot), fill = "cornflowerblue") +
coord_flip() +
labs(y = "Probability of being a bot",
x = "World leader",
title = "Probability of world leaders being a bot") +
theme_minimal()
So apparently we are almost certain Donald J. Trump isn’t a bot and very very nearly certain the Pope is a bot!
That’s all for this time, thanks for reading!
Credit: Photo by L B on Unsplash
