Alshaabi, T.1,2,3, Adams, J.1,2, Arnold, M.1,2, Minot, J.1,2, Dewhurst, D.1,2,4, Reagan, A.5, Danforth, C.1,2,3, and Sheridan Dodds, P.1,2,3
Science Advances (July 2021)
In real time, Twitter strongly imprints world events, popular culture, and the day-to-day, recording an ever-growing compendium of language change. Vitally, and absent from many standard corpora such as books and news archives, Twitter also encodes popularity and spreading through retweets. Here, we describe Storywrangler, an ongoing curation of over 100 billion tweets containing 1 trillion 1-grams from 2008 to 2021. For each day, we break tweets into 1-, 2-, and 3-grams across 100+ languages, generating frequencies for words, hashtags, handles, numerals, symbols, and emojis. We make the dataset available through an interactive time series viewer and as downloadable time series and daily distributions. Although Storywrangler leverages Twitter data, our method of tracking dynamic changes in n-grams can be extended to any temporally evolving corpus. Illustrating the instrument’s potential, we present example use cases including social amplification, the sociotechnical dynamics of famous individuals, box office success, and social unrest.
1 Vermont Complex Systems Center
2 Computational Story Lab, University of Vermont
3 Department of Computer Science
4 Charles River Analytics
5 MassMutual Data Science
