Language Selecting and Filtering Sample Clauses

Language Selecting and Filtering. This data collected from keyword search first goes through a language filtering step, as shown in Figure 3.2. In the Twitter database, there’s an item called “lang” (short for “language”) for each tweet that indicates the primary language that the tweet is written in. For English tweets, they have “en” for “lang”. To reduce noise caused by words appearing in different forms of multiple languages, this study filters the collected tweets by language, only keeping those written in English. The number of tweets kept in the final dataset is shown in Figure 3.3. Statistics in Figure 3.3 confirm that the filtered dataset still provides an enormous amount of social media data, ranging from 54k tweets/day to 86k tweets/day. As shown in Figure 3.3, there are some striking increases of tweet count on March 4th, March 8th, and March 12th. Although our research focuses on applying sentiment-based event detection, intuitively, these sudden increase in the number of tweets also indicate possible COVID events. Thus, we add Figure 3.2: Proposed Methodology of Preliminary Data Filtering. Figure 3.3: Number of Covid-19-related English tweets posted from March 1st to March 12th, 2020.