Twitter Dataset. From September 25th, 2016 to November 8th, 2016 about 135.5 million tweets were collected for this project.1 This was achieved by developing a client which took advantage of Twitter’s streaming application programming interface (API).2 The client watched for any public status update posted during this 1The client crashed three times during the course of collection, thus any tweets collected on those days were excluded from this project. 2Twitter Streaming API: xxxxx://xxx.xxxxxxx.xxx/streaming/overview time period which included any one of the following words: hillary, clinton, hillaryclinton, donald, trump, realdonaldtrump, election, and debate. It is true that these 8 words do not encompass every permutation of how those on Twitter might reference the two presidential candidates. Yet, for the purposes of this research it was important to anthologize the least skewed portrait of the online conversations about Clinton and Xxxxx. Therefore, the decision was made to only collect the formal names (i.e. their first and last names) that are used to reference either candidate. Because of how usernames are used on Twitter, as described supra, both candidates usernames were also included in collection. Additionally, the terms election and debate were also chosen as signal words for an important tweet to capture by the streaming client. The client only kept a status with either of these words when it appeared in conjunction with one of the other signal words, she, or he. The purpose of this is to capture tweets indirectly referencing either candidate. After the client compiled each new status into the corpus, it was classified as referencing Clinton or referencing Xxxxx. For the experiments using this data to be reliable indicators of the electorate’s attitude, this step was crucial. Because of this, criteria for inclusion into the subset of either candidates tweets was rigorously constructed. Damerau-Levenshtein distance was used to perform approximate string matching between a pre-determined set of terms for each canidate to determine if the tweet exclusively references one of the candidates. Through experimentation, an edit distance of 2 was chosen as the cutoff for matches. The resulting subset discussing Clinton included 35.7 million tweets and the subset discussing Xxxxx included 51.9 million tweets.
Twitter Dataset. As shown in Figure 3.1, twelve days’ tweets (in total 1,160,591 tweets) are scraped via the Twitter API using keyword search. Figure 3.1: Number of Covid-19-related tweets collected by keyword search, posted from March 1st to March 12th, 2020, without any language filtering. To ensure the dataset covers as many coronavirus-related posts as pos- sible, we manually read many tweets discussing the pandemic and collect relevant words that frequently show up in tweets under this topic. As a result, multiple keywords are included in the keyword search process: coronavirus, koronavirus, corona, wuhancoronavirus, wuhanvirus, kungflu, epidemic, covid- 19, covid19, corona virus, covid, chinavirus, and pandemic. Many prior studies rely on hashtag extraction to collect tweets containing the token of interest [12]. To explain the difference between hashtag extraction and keyword extraction, we would first need to understand the function of hashtags in Twitter. The hashtag symbol “#” is put before words or phrases to help Twitter users indicate the relevant topics of their tweets. Additionally, when people are interested in certain subject, they can simply tap on the hashtagged word or phrase to see all other tweets that have used the same hashtag. Here is an example of event detection using hashtag search: Xxxxxxxxx notices that people will use the hashtag “#oscars2010” to discuss related matters, because it’s an annual event and the use of hashtags indicates that the event is not only attractive to the public but more importantly, rela- tively rare [12]. However, since hashtags are usually used to highlight sudden heated topics, as the epidemic escalates in the U.S., people are less likely to use hashtags when discussing Covid-19-related subjects. In this context, hashtag extraction method increases the risk of undersized sampling. There- fore, to avoid any bias caused by the normalization of the pandemic, this study employs keyword search rather than hashtag extraction. In addition, some chosen keywords are in their misspelled form (e.g., koronavirus) or conveying racist connotations (e.g., kungflu and chinavirus). Although these words rarely appear in mainstream news reports, they are frequently used by social media especially in the early stage of coronavirus pandemic. The use of these racist words has become even more prevalent after President Xxxxx blames China for not restricting international flights and eventu- ally infecting the whole world, inciting the ethn...
