by Nastasia Griffioen, Marieke van Rooij, Anna Lichtwarck-Aschoff, and Isabela Granic
Published onJun 17, 2020
Toward Improved Methods in Social Media Research
·
Abstract
Both academic and public interest in social media and their effects have increased dramatically over the last decade. In particular, a plethora of studies has been conducted aimed at uncovering the relationship between social media use and youth wellbeing, fueled by recent concerns that declines in youth wellbeing may well be caused by a rise in digital technology use. However, reviews of the field strongly suggest that the picture may not be as clear-cut as previously thought, with some studies suggesting positive effects and some suggesting negative effects on youth wellbeing. To shed light on this ambiguity, we have conducted a narrative review of 94 social media use and wellbeing studies. A number of patterns in methodological practices in the field has now become apparent: self-report measures of general statistics around social media use dominate the field, which furthermore often falls short in terms of ecological validity and sufficient use of experimental designs that would enable causal inference. We then go on to discuss why such practices are problematic in some cases, and more importantly, which concrete improvements can be made for future studies that aim to investigate the relationship between social media use and wellbeing.
Keywords: social media, methodology, wellbeing, objective data, self-report, qualitative research
Disclosure and Acknowledgment: The authors declare that the research was conducted in the absence of any commercial or financial relationships that could be construed as a potential conflict of interest. This work was supported by the European Research Council (ERC) Consolidator Grant [683262].
Open Access License: This work is licensed under a Creative Commons Attribution-NonCommercial-NoDerivatives 4.0 International License (CC- BY-NC-ND). This license permits copying and redistributing the work in any medium or format for noncommercial use provided the original authors and source are credited and a link to the license is included in attribution. No derivative works are permitted under this license.
Disclaimer: Interactive content is included in the online version of this article.
Contact Information: Correspondence concerning this article should be addressed to Nastasia Griffioen, Behavioural Science Institute, Radboud University, Montessorilaan 3, 6525 HR Nijmegen, The Netherlands. Email: [email protected]
Introduction
As profoundly social beings, humans crave social interaction to the extent that lack thereof affects us negatively in all kinds of ways. People experience reduced stress when social support is abundant (e.g., Cohen & Wills, 1985) and are at increased risk of death when social ties are scarce (Berkman & Syme, 1979). The importance of social connections is also illustrated by the incredible popularity of social media. There are social networks available in almost any domain of our lives (e.g., dating—Tinder; careers—LinkedIn; games—Discord). The most popular networks are not just used by many; they are used by most: In the United States alone, roughly three-quarters of the population are members of one or more social networks (Pew Research Center, 2018a), and almost 3.48 billion people worldwide are actively using social media (Kemp, 2019, January 30). Zooming in on the younger generation, we see an astounding 94% of 18–24-year-olds and 85% of 13–17-year-olds (Pew Research Center, 2018) reporting using an online social medium.
The fervent adoption of these platforms, especially among the younger generations, has sparked interest as well as concern, primarily among parents and schools. Many worry that teenagers are “glued to their phones” (e.g., Zolfagharifard, 2017, August 31), paying less and less attention to the physical world around them. Indeed, children’s and adolescents’ lives look different from those of their parents. The activities they engage in may seem odd, and perhaps detrimental, in the eyes of the older generations that did not grow up using smartphones and tablets. Some studies even seem to suggest that current teenagers prefer to connect with peers through their phones rather than in person (Common Sense Media, 2018). Digital technologies, such as social media, do not just “complement” previous ways of communicating, but have instead replaced their analogue counterparts almost entirely, as is the case with emails almost wholly replacing letters (Schmid, 2011, October 3). Some even argue that as a result of such an attachment to mobile media, the majority of UK children spend less time outside than prison inmates do (Carrington, 2016, March 25).
Following a growing concern among the general public, research on the topic of social media in the last decade has increased as well. Findings on the relationship between social media use and well-being are, however, far from straightforward. Recently, a number of reviews have been published, and one thing most of them have in common is the fact that conclusions regarding the effect of social media use on well-being seem hard to draw (e.g., Erfani & Abedin, 2018). One review even concluded that the field of social media and well-being research is dealing with “contradictory evidence, while revealing an absence of robust causal research regarding the impact of social media on mental wellbeing of young people” (Best et al., 2014, p.1).
The field’s ambiguity is puzzling, but the ways in which the field has conducted its studies so far are likely to offer an explanation: Research methodologies are not just a tool to perceive and assess, but instead play a large role in what is perceived and how it is assessed. This article aims to give an overview of the research practices currently being applied in social media use and well-being research and shed light on the implications of study design choices. We will discuss the dominant presence of self-report and the types of data that are gathered, as well as the importance of ecological validity and causality. A clear sense of the way in which social media and well-being research is currently being conducted should be instrumental in facilitating the field to build more reliable, robust, and informative studies addressing the link between social media use and well-being. Our synthesis of methodological trends will be followed by a brief discussion of the strengths and weaknesses of these methodologies as well as suggestions for improvement that will enable us to answer questions like “What do adolescents do on social media?” “What do adolescents expect from social media?” “How do they feel when interacting on social media?” and “Why do they use social media at all?”
Current Research Practices
Literature Search Specifications
The search for articles on the topic of social media use and well-being was conducted in three different databases—PsycINFO, Web Of Science, and MEDLINE. Using the search queries “social media use” AND (“wellbeing” OR “well-being”) and restricting the search to 2010–2018, we found a total of 129 articles. To investigate the studies most relevant to the current social debate around social media use, we focused on general, healthy population samples, and thus removed studies focusing on disordered samples (e.g., schizophrenia or attention deficit hyperactivity disorder samples). Duplicates, non-peer-reviewed, nonempirical, and/or non-English articles were also excluded. Cross-checking the reference lists of (review) articles with our selection brought the total number of articles in our review to 72. During the time of writing, eight more articles were included (up to June 2019) and three more articles were added during the review process, bringing up the total number of articles included in our review to 83. Because some articles feature more than one study, the total number of empirical studies included in our review is 94. See Figure 1 for a schematic overview of the search process.
Patterns in Research Practices
First, we found that the field is dominated by studies relying on self-report measures. Of the reviewed studies, 81.9% (n = 77 out of 94) quantified social media use by asking participants to retrospectively report on their social media use. Only 6.4% of the studies (n = 6 out of 94) used some form of objective assessment of social media use, based, for example, on Facebook’s activity logs (e.g., Burke et al., 2010) or other types of recordings of people’s activity on social media (e.g., Verduyn et al., 2015).
When looking closer at the types of self-report measures (i.e., questionnaires) of social media use, it is striking that many studies either seem to use questions developed by the researchers for the first time in their particular study, or use selected (and often adapted) questions from already available scales such as the Facebook Questionnaire (Ross et al., 2009). The questions used are most often along the lines of “On average, how much time per day/week do you spend on Facebook/social media?” and “How often do you do the following things on social network sites?” (listing a number of activities such as posting and chatting). This is in line with our observation that mostly general measures of social media use are collected (see below). Few studies used a preexisting, validated questionnaire, the most frequently used being the Facebook Intensity Scale (n = 7 out of 79) (Ellison et al., 2007).
Other scales that have been used include the Instagram Activity Scale (Yang, 2016; used by Yang & Robinson, 2018), the Multidimensional Scale of Facebook Use (Frison & Eggermont, 2015; used by Frison & Eggermont, 2015); (Faelens et al., 2019), the Bergen Facebook Addiction Scale (Andreassen et al., 2012; used by Dhir et al., 2018), the Bergen Social Media Addiction Scale (Andreassen et al., 2017; used by Worsley et al., 2018), the Facebook Questionnaire ((Ross et al., 2009); used by (Ryan & Xenos, 2011); (Simoncic et al., 2014), the Social Networking Survey (Davila et al., 2012; used by Davila et al., 2012), the Multidimensional Facebook Intensity Scale (Orosz et al., 2016; used by Phu & Gow, 2019), and the Facebook Activity Measure (Shaw et al., 2015; used by Shaw et al., 2015). As becomes apparent from this list, many of these scales are used in only one or two studies, and there is thus little overlap between the studies in terms of the actual scales that are used. Similarly, for single questions there seems to be little consensus on wording, even though many of these questions aim to gauge the same thing, for instance, time spent on social media. This understandably makes it difficult to draw conclusions across studies and may play a role in explaining why superficially similar studies produce different findings.
Moreover, almost three in five studies focused solely on general measures of social media use (n = 55), such as the frequency of use, preferences for certain social media platforms, and average time spent on social media per week. In contrast to general measures of social media use, some have suggested that the way in which social media are used is of consequence to its effects; specifically, active use is defined as posting, commenting, and sharing status updates, whereas passive use is defined by a more uninvolved consumption of social media content, for instance, by browsing and scrolling (Thorisdottir et al., 2019). Indeed, out of the 39 studies that looked at more specific types of use and/or experiences, roughly a quarter (n = 11) differentiated between active and passive uses of social media, and three-quarters (n = 28) looked at more specific social media behaviors (e.g., posting), but not necessarily with a focus on the active versus passive dichotomy. Within this group of 39 studies, a group of articles (n = 4) collected specific metrics such as frequencies per social media activity type (Kim & Kim, 2017), but aggregated them into general metrics for subsequent analysis. It is thus clear that relatively little attention is being paid to the details of social media use, despite recent indications (Burke & Kraut, 2013; Burke et al., 2011; Huang, 2010) that this may be exactly what is needed to get a better understanding of what is going on and move the field forward.
In line with the fact that the vast majority of studies rely on retrospective accounts of general measures of social media, we have found that only a small proportion of the studies, about 18.1% (n = 17 out of 94), attempted to incorporate (parts of) the experience of social media use in their studies. In some of these cases, study design elements were less than ideal from an ecological validity point of view. For example, in some studies, participants were overtly restricted in the ways they could behave, for instance, by prohibiting participants to share posts on Facebook (Tobin, et al. 2015; for other examples of restriction, see Deters & Mehl, 2013; Yuen et al., 2019; Verduyn et al., 2015). Other studies clearly manipulated expectations, for instance, by telling participants to expect comments on their posts from coparticipants (thus rendering the fact that participants felt bad when these comments remained absent not particularly surprising, see Tobin et al., 2015). In some studies, manipulations may have been less effective than intended given the aim of the manipulation. For instance, in a study by Vogel et al., 2014, an (intended) downward social comparison target (i.e., someone who is perceived as inferior to the self) may well have been interpreted as an upward social comparison target (i.e., someone who is perceived as superior to the self) by some participants. In sum, truly realistic implementations of social media use are still rare in the present body of literature. This is problematic because it is important that we draw conclusions based on examples that accurately reflect real life.
Last, we only found 17 studies (18.1%) that used an experimental setup to examine the link between social media use and well-being. The remaining 88.3% (n = 77) used observational methods rather than experimental ones. This is a serious limitation given that strong claims are being made about the detrimental effects of social media use in the popular media (e.g., Barr, 2019, October 10; Twenge, 2018). One notable subgroup in these observational studies used an experience sampling methodology (ESM) (n = 5) (Csikszentmihalyi & Larson, 1987) in which participants’ social media use and well-being were assessed through self-report multiple times a day over an extended number of days (Kross et al., 2013; Steers et al., 2014; Verduyn et al., 2015; Wenninger et al., 2014). Another 4 out of these 77 studies used a longitudinal design (Booker et al., 2018; Frison & Eggermont, 2015; Heffer et al., 2019; Matook et al., 2015), which is an important step toward the ability to draw conclusions about causality, but still surprisingly rare in the body of studies that we have reviewed. See Figure 2 for a summary of the methodological patterns discussed.
Thus, we have found methodological patterns emerging from the literature that can be summarized as follows: The field predominantly relies on self-report measures of general statistics around social media use, and often falls short in terms of ecological validity and sufficient use of experimental research that would enable causal inference. For an overview of the studies that were reviewed and their methodological characteristics, see Table 1. Now that the general methodological landscape of social media use research has been painted, we move on to address its features in more detail and look ahead to the horizon of future research by offering suggestions for improvement in studies yet to come.
Pitfalls and Solutions in Current Research Practices
Self-Report Data
The first and perhaps most evident pitfall currently present in the field of social media use and well-being research is its overreliance on self-report data. Self-report continues to be, by far, the dominant measure in social media research. Of course, psychology as a field has been using self-report from its inception and has been discussing its merits (e.g., understanding people’s own perception of their behaviors) and limitations (e.g., biases, social desirability, single-subject shared variance, and so on; e.g., Rosenman et al., 2011; Allport, 1927) for equally as long. Thus, the problem and ubiquitousness of self-report measures is not particularly unique to social media research. In fact, it is curious that our 6.4% of studies that measure actual behaviors in social media use closely resemble Doliński's (2018) observation that more generally, across the field of personality and social psychology, behavioral measurement was equally rare, with only 6% of studies including behavioral measures in their broader review.
While issues with self-report are thus not new, recent studies have highlighted the severity of the problem when using this type of data to make any inferences about social media’s causal relations to well-being. Specifically, recent studies indicate that there is a low correlation between people’s subjectively reported time spent on phones and objective data extracted from the phones themselves (Boase & Ling, 2013; Ellis, 2019). Similarly, the overlap between psychometric scales measuring phone use and objective behavior is “generally poor,” with some correlations as low as .2 (Ellis et al., 2019).
This may not seem surprising in light of long-lasting debates around the merits and limitations of self-report, but the implications are vast: Decisions around social media use—and more generally screen time use—are being made on all sides. Parents panic about their children’s well-being, a panic that is fueled by mostly correlational studies that suffer from the problems outlined above. Policy decisions are being made (e.g., in China, see “China province to ban homework,” (2019); “China to impose curfew,” 2019) that affect millions of children and teenagers, while real concerns about (mis)use remain unaddressed. For example, children may be visiting harmful websites or being irresponsible about what data they share when they are online but are likely reluctant to share such details when asked. It is thus safe to say that real-world decisions are being made at a large scale, and we as a field need to make sure that the information at the root of those decisions is reliable and valid. It has become clear now that if we want reliable data to answer questions like “Does time spent on social media relate to lower well-being?” we likely need to look to sources other than self-report.
It is remarkable how few studies have leveraged the digital nature of social media when gathering quantitative data regarding social media use. The devices on which social media are accessed gather and store large amounts of (in principle, objective) data pertaining to the activities being carried out on them (Piwek et al., 2016). The use of such data is precisely one of the solutions we propose here. Recently, social media companies have made it easier for their users to access their own data. Facebook, Instagram, Twitter, and Snapchat now all provide their users with the option to download their data. See Table 2 for an overview of the “Download My Data” functionality offered by each of the social media platforms. Since then, several studies in the field of social media use and well-being have exploited the opportunity to access personal logs (e.g., Burke & Kraut, 2016; Park et al., 2016; Shakya & Christakis, 2017), but they remain few and far between. In addition to a downloadable “data dump” and other features such as Facebook’s “Activity Log”—which contains a slightly different array of variables (e.g., likes from others directed at your own posts, and who follows you)—some information may be available within the social media app itself. By taking screenshots or screen recordings of the social media app that is being used, relevant social media behaviors and information can be assessed in an objective manner (e.g., what sorts of posts are most prominent in a user’s news feed: those by friends or those by companies and pages?).
In some cases, however, none of these options are sufficient or appropriate for the desired study design. For instance, if assessment of social media use needs to dynamically interact with other elements of the study, after-the-fact data logs will not work. This could be the case when researchers want a questionnaire on the phone to be triggered by the participants’ starting to engage with a social media app, or if the researchers want a message for the participant to pop up after a certain amount of time has been spent on a specific social media platform. One way to solve this problem is to leverage “application programming interfaces” (APIs) offered by social networks that allow third-party apps to interact with information from these networks (Lomborg & Bechmann, 2014). Such data connections between a social network and other companies are often used to achieve targeted advertisement. APIs also enable the creation of custom-made solutions, such as apps that serve as a portal through which participants will use their preferred social platform for the duration of a study. An environment could be created that mirrors the social media platform(s) of interest, forming a hub where all platforms come together, while allowing for the collection of specific data that are not otherwise accessible (such as timestamps for when a social network app was opened). This way, the researcher can continuously (and dynamically) gather information about everything the user does and sees. In addition, communication can be set up between the portal app and another system to, for instance, trigger questionnaires or manipulations.
That being said, a recent article by John & Nissenbaum (2019) has pointed out that certain aspects of social media use such as “disconnectivity” (i.e., actions such as “unfriending” somebody or “unliking” content) do not seem to be well represented within APIs. When exploring the use of APIs for research purposes, it seems important to determine if these APIs will be accessible to public researchers and able to deliver the necessary information. While all the suggestions that we have discussed here may have downsides such as effort or cost, overcoming these significant obstacles seems critical for the sake of rigorous and reliable science if more convenient and low-cost options are insufficient or inappropriate for the study at hand.
We would like to mention, however, that more (and better) data comes with its own potential pitfalls and ethical considerations, such as those of privacy. Social media use data—and in particular the data that we propose is of most value—is often highly sensitive and might contain information not only about the participant who has given consent for its use, but also about the participants’ social ties whose consent is not obtained. It is of vital importance to make sure that the privacy of all parties involved is guaranteed, or if anonymization for whatever reason is impossible, at least that full transparency toward participants is observed (e.g., specifics about the type of data that will be gathered, what it will be used for, and who will be able to access it). These considerations have become increasingly complex in light of the relatively recent developments toward procedures that promote open science. Sharing data openly in repositories, for instance, is directly at odds with the privacy of participants when it comes to sensitive data. This issue is complex and deserves a deeper analysis of the balance between risks and mitigations, but they are currently beyond the scope of the present work. There are, however, important strides being taken toward a coherent strategy to navigate these security and privacy matters (Dennis et al., 2019).
Specificity of Social Media Use Data
The second pattern that has emerged from our review of the literature revolves around the level of detail with which social media use has been investigated so far. In light of the inconclusiveness of the current body of findings, some have suggested that general metrics and the global, increasingly vague concept of “screen time” need to be replaced with designs, methods, and analytical techniques that can concretely differentiate between different kinds of social media activities (Burke et al., 2010). Social media contain an immense range of functionalities, and the experiences people have when interacting with social media can be extremely diverse. What social media mean and offer to their users has been developing rapidly ever since they appeared. Classmates.com, the first social network, was an instant hit when it launched in 1995, but its features—which initially consisted of being able to simply, and only, track down school yearbooks—will seem unsatisfactory to modern-day social media users. Many more “sharing” functions have now been added to the online social media arsenal, such as the sharing of music, feelings, activities, locations, friends, photos, and even belongings. Users are able to share almost all and any aspects of their lives while viewing often carefully curated snapshots of the lives of others. Not taking this diversity into account means losing sight of information that could explain differences in social media’s effects.
We propose that behaviors and experiences should be viewed on a more specific, and thus functional, level. For instance, by distinguishing between actions that are “active” versus “passive”, or distinguishing between the effects of viewing posts that come from different types of sources (e.g., friends versus celebrities), researchers can delve deeper into the rich nature of social media in attempts to determine why and when social media may affect well-being. To help researchers navigate the diversity of social media behaviors, we created an overview of the functionalities that social media users can engage in on the most-used social media platforms (see Table 3).
When designing future studies investigating social media use, it is similarly important to be aware of selection biases pertaining to the platform being studied. For instance, while Facebook has dominated the social media landscape for a long time, teenagers aged 13–17 have, to a large degree, abandoned Facebook in favor of Instagram and Snapchat (Pew Research Center, 2018b). It is therefore important to ask questions such as “Do users of different platforms differ in meaningful ways?” and “Is there something about the users’ goals that leads them to use different social media?” Whether such shifts in platforms’ user bases are problematic for a study’s design naturally remains to be assessed by researchers individually.
It should be noted, however, that attention to more and more fine-grained details of social media use alone will not do. A greater attention to detail paradoxically also entails that we pay attention to the larger behavioral patterns that surround social media use, so that we may understand the contexts in which these media are being used, separately and in parallel. Young people do not just use one app; they use tens of different apps, sometimes at the same time, and we need to be able to capture this variety of use to better understand the entire digital ecosystem and users’ connections to it. This, too, requires that we gather fine-grained and objective data, for instance, through the use of aforementioned APIs or screen recordings (Reeves et al., 2019; Ram et al., 2020).
Last, being specific about what children and teenagers do on social media and how it makes them feel also requires probing their subjective experiences and getting as close as possible to their actual lived experiences. Combining the strengths of the reliability of objective data with the depth and sensitivity to context afforded by subjective approaches is a challenge because it would require the integration of multiple methods. Nevertheless, the first steps toward such approaches have already been made, for instance, by Piwek & Joinson (2016), who have investigated how and with whom Snapchat is used by adolescents. One promising avenue to further this direction of research along is through the use of “stimulated recall.” Bloom (1953, p. 161) expressed that the primary aim of the method is “that the subject may be enabled to relive an original situation with vividness and accuracy if he is presented with a large number of the cues or stimuli which occurred during the original situation.” Stimulated recall is an approach in which the benefits of quantitative research (i.e., attention to context, motivations, and subjective experience) are supported by objective data. Regular retrospective self-report regarding behaviors (or even feelings) is—as we have seen—a risky business given the difficulty people have with accurately recalling past events. Stimulated recall relies on recall immediately following the event of interest. Participants are supported when recalling relevant aspects of this past experience through the use of materials such as audio and/or video recordings and physiological data. Such methods have often been applied in educational sciences (e.g., Calderhead, 1981; De Witt, 2008; Meier & Vogt, 2015) and in user experience research to systematically assess what users think and feel during certain actions or events.
A concrete example in the context of social media use research may be useful for demonstrating the power of the stimulated recall method. We recently used this methodology in a study in which we asked the participants to wait in a room for a short amount of time (10 min), during which we collected video footage of their actions. Following the waiting/monitoring period, we informed the participants about the real aim of the study (i.e., mapping out what adolescents do on their phones and on social media in particular, for which reasons, and how it makes them feel). If participants consented, we proceeded with a stimulated recall interview phase. During this phase, we used the video footage of their activities as well as in-app logs to help the participants answer a number of structured questions about their phone and social media use during the waiting period (for a more detailed description of this particular implementation, see Griffioen et al., 2020). By implementing a highly structured stimulated recall interview in combination with objective data retrieved, this methodology allows researchers to address the current lack of reliable, objective information in the field. It also helps us focus more on the content, function, and processes of social media use and provide a structured way of gathering these data.
Ecological Validity
A third and related problem pervading current social media and well-being research pertains to its ecological validity. When studying a behavior that occurs in day-to-day life as prominently and frequently as social media use, it is important to make sure that the context in which it is studied reflects the character of these everyday situations. This is especially important when laboratory studies on this subject are conducted because these contexts are most dissimilar from everyday life. Most importantly, social media are steeped in perpetually social, personally meaningful, and emotionally salient contexts. These contexts, however, are rarely investigated, and the focus so far has lied predominantly on the technology itself, not on its function for its users: Only 17 of the many experiments that we reviewed attempted to take social and emotional contexts into consideration and many of these studies have serious limitations (see section “Patterns in Research Practices” for examples). The key elements of the experience and use of social media are often overly controlled or even overlooked in laboratory experiments, even though it is essential to keep the central, functional feature of social media use in mind: They are fundamentally social platforms with social interaction and relationships as their key purpose. While field research is an important avenue for ensuring that the context of measurement is ecologically valid, laboratory studies are nevertheless sometimes required to assess causal links between social media use and, for instance, aspects of well-being. The discussion here thus revolves not only around increasing ecological validity by conducting field studies, but also by conducting laboratory studies in a better, more context-sensitive way. Ensuring that laboratory social media experiences reflect real-life use (i.e., that they are ecologically valid) requires that we improve our understanding of what it feels like to use social media, both in the moment and in past experiences, with the goal of incorporating those key elements in laboratory re-creations.
First, to understand what social media use evokes in the user in the moment, it is important to acknowledge that there is a tremendous amount of salience tied to social information. This is unsurprising because we are social creatures, and rely, to a large extent, on other people (Baumeister & Leary, 1995; Beckes & Coan, 2011; Berkman & Syme, 1979; Bloomberg et al., 1994; Cohen & Wills, 1985). The curated nature of social media further augments this salience because users receive information from sources that are important to them, be they close friends, family, or celebrities. Indeed, the social salience of social media has been previously acknowledged: Social media are infamous for their role in eliciting social comparison (Appel et al., 2015; Chow & Wan, 2017; Fardouly et al., 2015; Haferkamp & Krämer, 2011; Jang et al., 2016; Nesi & Prinstein, 2015). When the social salience experienced in social media is absent from its re-creations, a study no longer provides insight into the real-life processes related to social media use.
One way to investigate the determinants of social media salience is to assess arousal, for instance, through physiological measures such as galvanic skin response (GSR) (Bach et al., 2010), pupil dilation, heart rate (Bradley et al., 2008; Wang et al., 2018), or eye-tracking in combination with facial expressions, when people are viewing social media content. Through structured and thorough debriefing afterward (e.g., by implementing a variation of the stimulated recall method discussed earlier), it is possible to assess which feelings and thought processes are being evoked by the information that one encounters on social media (e.g., “I saw a post by a friend dedicated to her mother, and I felt happy and sad at the same time”), and for which reasons (e.g., “I felt happy and sad because that was something I went through myself, and I recognized myself in her story”). Such an understanding of the emotions and thoughts taking place during social media experiences is essential if we want to be able to re-create these experiences in laboratory settings, and thus create ecologically valid research contexts.
Second, the role of prior experiences and future expectations when using a medium in which a lot of social information is encountered is often overlooked. Participants are not blank slates; they have gained extensive prior experiences in the (online) social realm. We propose that for laboratory experiments around social media use to be most informative and ecologically valid, a participant’s prior experiences on social media need to be taken into account. Only then will we be able to meaningfully interpret and understand the ways in which participants respond to events in experimental social media contexts, and why they do so. Given that prior experiences will inform future expectations, well-being is likely related to these experiences and expectations for what the future will bring. Feelings such as anxiety and depression, for example, are marked by a negativity bias regarding future events (Korn et al., 2014). Since much of social media use research is related to its effects on well-being, it is striking that fairly little attention has so far been paid to people’s subjective experiences, motivations, and expectations when using social media.
In social media research, assessing what such prior beliefs or expectations look like can be as simple as asking participants what sort of information they expect to see, how they expect to feel, and why. These expectations may or may not be related to participants’ self-evaluative beliefs, and thinking about how their beliefs are updated throughout social media interactions may be informative in investigating the link between their prospective social media use and well-being. Information about such “priors” can, for instance, help us understand why in some individuals we seem to find detrimental effects of social media, while we do not in others. Such methods—to our knowledge—have not yet been implemented in social media research, but there is interesting research in adjacent fields that might offer different ways of thinking about how we can assess sequential social learning processes and what shape they take.
In a study by Will and colleagues (2017), for instance, computational models helped determine that the way in which we update our self-evaluative beliefs is similar to the way in which we learn about others. Similar models could be applied to social media use research to form and test predictions about how contact with and processing of different salient aspects of social media (e.g., content of posts, or types of social ties encountered) might change expectations about future social media visits. These expectations, in turn, might affect the extent to which users internalize social media content during those future visits (i.e., the extent to which social media end up affecting their well-being). In sum, forming models—be they conceptual or formal—of social media experiences at longer temporal scales will provide us with a better understanding of real-life social media use and its relationship to well-being.
Causality
In addition to objective data and ecological validity, the ability to test causality is important to draw conclusions about the effect of social media on well-being. As we have discussed, however, the literature is mostly dominated by observational designs, which—similar to self-report measures—can be insightful if implemented appropriately. Observations enable us to study people in real, everyday situations, thus providing the opportunity to uncover behaviors or phenomena that would otherwise remain unnoticed (Allen, 2017). However, observational designs have one major drawback: They do not allow for causal inference. Consequently, experimental or (semi-)longitudinal designs are important to provide us with information about whether a relationship might be causal.
In our review of the empirical literature, we found only nine studies, four longitudinal and five ESM studies, that have attempted to circumvent the primary downside of observational designs. More experimental designs are needed, and researchers designing these studies will benefit from understanding participants’ prior and current experiences and expectations, if manipulations are to be effective and realistic. In particular, ESM (Csikszentmihalyi & Larson, 1987) offers interesting avenues for approximating some sense of causality while still maintaining more of the ecological validity often found in observational studies. Through repeated measures of the constructs of interest (e.g., daily stress), ESM provides a way to minimize response biases while still measuring variables of interest within the participants’ natural environment (Riediger, 2009). Although usually no experimenter-induced manipulations take place during the ESM period, the extended nature of the measurements also allows for tracking of the order in which certain events have taken place, which helps form a sense of which events in a person’s life (e.g., stressful experiences) have an effect on other elements of their life (e.g., mood).
Modern ESM studies leverage the fact that most people carry around smartphones on which they can receive texts, emails, and links to websites containing questions, and through which data can be saved directly to a secure database. No wonder, then, that there is a growing number of studies that implement ESM to gather data about people’s well-being on the same device that is their portal to the digital social world (e.g., Steers et al., 2014; Verduyn et al., 2015; Wenninger et al., 2014). While ESM is promising, and researchers are continuously working hard to improve ESM reliability (e.g., Berkel et al., 2020), there are a number of adjustments that we feel that can be made to improve the quality of research given the nature of social media use.
Social media are being used often at many different moments during the day, and some research suggests that a portion of this use may be happening almost subconsciously (Lin et al., 2015; Montag et al., 2015), for instance, when waiting for a bus, swiping around on one’s phone, looking for something to read or do. Thus, even if you are asked about your use and experiences in social media five random times a day, you are likely to have a hard time remembering what it is exactly that you did or saw when you last visited a social network on your phone. To further minimize this recall bias, we suggest that ESM measurements in future social media studies could be triggered by specific events such as the use of social media itself (for instance, immediately following the closing of a social media app). Although such event-triggered ESM methods do not seem to have been implemented yet (even outside the field of social media research), we propose that they are a critical improvement on the traditional random-measurement approach implemented by most ESM studies. We further argue that—given our suggestions on gathering objective data regarding social media, in particular regarding the use of APIs—such event-based triggers are feasible and will further improve our ability to draw meaningful conclusions from ESM data.
In addition, we urge researchers who are implementing ESM in their study of social media and well-being to include objective measures of what it is that participants do and see (on social media) in their analyses (rather than only using such information to trigger event-based questions). Such data, in contrast to the data necessary for event-based triggers, could be collected retrospectively using the data logs that were mentioned in section “Self-Report Data”. This way, ESM questions can be aimed at assessing the qualitative side of people’s social media use (e.g., “How did you feel while reading other’s posts on social media, and why?”) when these experiences are most “fresh”, whereas objective data can tell us what it is exactly that people were doing and how often/for how long. The combination of objective measures of use and/or information encountered on social media and well-timed assessments are a promising avenue that needs to be explored. Ultimately, such methods can allow researchers in the field of social media use and well-being to find the answers not only to questions like “How do particular social media experiences relate to later mood and well-being?” but also to questions like “Do adolescents use social media differently depending on their mood?” “Do adolescents who feel depressed search for regulating social experiences on social media?” and “Does social media use elevate feelings of anxiety and stress or does it help regulate those feelings?” These are the questions that are at the core of social media use and well-being research.
Conclusion
While there is a lot of attention to (and concern about) social media and their effects, the link between social media use and well-being is far from well understood. To shed light on the state of the social media use and well-being literature, we synthesized the methodological characteristics of empirical studies conducted since 2010. In our literature review, we identified patterns that are present in this field which require improvement and adjustment to the still relatively new and poorly understood context of social media. Unrealistic and highly artificial research contexts are often the default designs in the field. Observational studies that lack sufficient ecological validity and the possibility of causal inference are abundant, whereas experimental work is scarce. Moreover, the function of social media use and specific ways in which that use addresses users’ goals are understudied, and self-report seems overused even though these reports are poorly related to objective measures.
There is a need for improvement of the research methodologies applied in this field, especially given the great weight assigned to studies examining the link between social media use and well-being. As with most new technological phenomena, a great deal of suspicion has been formed regarding what the use of social media does to the mental well-being of children, teenagers, and young adults. Policy changes (e.g., see World Health Organization, 2019 for guidelines issued regarding general screen time), clinical classifications (e.g., the ICD’s and DSM’s potential inclusion of internet addiction; Poli, 2017), and parenting guidelines (e.g., Elmore, 2018, March 15) are being founded on a body of literature that we have demonstrated is not yet strong enough to bear the burden of proof for these large-scale implementation strategies.
However, there is a substantial number of ways in which these improvements can be made. Staying as close as we can in our studies to the real experience of what it means to interact with others on social media is of paramount importance. In addition, there is room for a lot more specificity in research into social media and ensuring that objective and reliable data gathered are all research goals that can be achieved in the future of social media use and well-being research. We hope that this article provides researchers that are examining the link between social media use and well-being with some useful suggestions for how to implement methodological improvements. With methodological innovations that are becoming increasingly accessible to all researchers, we are optimistic that the new generation of emerging studies on social media use and well-being will provide powerful and timely insights into these complex relations.
Tables
Table 1: Methodological Overview of Reviewed Studies
Year
Author Indication
Specificity
Nature of data
Study Design
2010
Burke, Marlow, Lento
specific
objective
observational
2011
Farahani et al.
general
self-report
observational
2011
Haferkamp, Krämer
general
not clear if manipulation was checked
experimental
2011
Haferkamp, Krämer
general
not clear if manipulation was checked
experimental
2011
Lee, Lee, Kwon
general
self-report
observational
2011
Ryan, Xenos
general and specific
self-report
observational
2012
Deters, Mehl
specific
objective
experimental
2012
Chou, Edge
general
self-report
observational
2012
Pantic et al.
general
self-report
observational
2012
Davila et al.
general
self-report
observational
2012
Davila et al.
general
self-report
observational
2013
Jelenchick, Eickhoff, Moreno
general
self-report
observational
2013
Ahn, Shin
specific
self-report
observational
2013
Kross et al.
general
self-report
observational: ESM
2013
Krasnova, Wenninger, Widjaja, Buxmann
general
self-report
observational
2013
Krasnova, Wenninger, Widjaja, Buxmann
specific: active vs. passive
self-report
observational
2013
Wang
general and specific
self-report
observational
2013
Apaolaza et al.
general
self-report
observational
2014
Vogel et al.
specific and general
self-report
observational
2014
Vogel et al.
general
not clear if manipulation was checked
experimental
2014
Lee et al.
general
self-report
observational
2014
Kim, Chung, Ahn
general
self-report
observational
2014
Labrague
general
self-report
observational
2014
Lee
general
self-report
observational
2014
Sagioglou, Greitemeyer
general
self-report
observational
2014
Sagioglou, Greitemeyer
general
not clear if manipulation was checked
experimental
2014
Sagioglou, Greitemeyer
general
self-report
observational
2014
Simoncic et al.
general and specific
self-report
observational
2014
Steers, Wickham, Acitelli
general
self-report
observational
2014
Steers, Wickham, Acitelli
general
self-report
observational: ESM
2014
Wenninger, Krasnova, Buxmann
specific and general
self-report
observational: ESM
2015
Frison, Eggermont
specific: active vs. passive
self-report
observational: longitudinal
2015
Tobin, Vanman, Verreyne, Saeri
specific: active vs. passive
self-report
experimental
2015
Tobin, Vanman, Verreyne, Saeri
specific: active vs. passive
self-report
experimental
2015
Koutamanis, Vossen, Valkenburg
specific
self-report
observational
2015
Verduyn et al.
specific: active vs. passive
objective
experimental
2015
Verduyn et al.
specific: active vs. passive
self-report
observational: ESM
2015
Vogel et al.
general
not clear if manipulation was checked
experimental
2015
Appel, Crusius, Gerlach
general
not clear if manipulation was checked
experimental
2015
Sampasa-Kanyinga et al.
general
self-report
observational
2015
Fardouly et al.
general
not clear if manipulation was checked
experimental
2015
Hayes, Van Stolk-Cooke, Muench
general and specific (collected, but aggregated)
self-report
observational
2015
Krasnova e tal.
specific (used as control variables)
self-report
observational
2015
Lin, Utz
general
self-report
observational
2015
Lin, Utz
general
not clear if manipulation was checked
experimental
2015
Matook, Cummings, Bala
specific: passive vs. active
self-report
observational: longitudinal
2015
Rae, Lonborg
general
self-report
observational
2015
Shaw, Timpano, Tran, Joormann
general and specific
self-report
observational
2015
Tandoc Jr., Ferrucci, Duffy
general and specific
self-report
observational
2016
Woods, Scott
general
self-report
observational
2016
Lin et al.
general
self-report
observational
2016
Levenson et al.
general
self-report
observational
2016
Shakya, Christakis
specific
objective
observational
2016
Hicks, Brown
general
self-report
observational
2016
Yang
specific
self-report
observational
2016
Burke, Kraut
specific
objective
observational
2016
Brusilovsky et al.
general
self-report
observational
2016
Van Zoonen, Verhoeven, Vliegenthart
general
self-report
observational
2016
Gerson
general
self-report
observational
2016
Frison, Eggermont
specific: active vs. passive as well as public vs. private
self-report
observational
2016
Jang, Park, Song
general
self-report
observational
2016
Park et al.
specific
objective
observational
2016
Pittman, Reich
general
self-report
observational
2016
Wood, Center, Parenteau
general
self-report
observational
2016
Tromholt
specific (active vs. passive) and general
self-report + not clear if manipulation was checked
experimental
2017
Kim, Kim
specific (collected, but aggregated)
self-report
observational
2017
Chen, Li
specific and general
self-report
observational
2017
Karikari, Osei-Frimpong, Owusu-Frimpong
general
self-report
observational
2017
Burrow, Rainone
general
self-report
observational
2017
Burrow, Rainone
specific
not clear if manipulation was checked
experimental
2017
Kim
general
self-report
observational
2017
Twenge
general
self-report
observational
2017
Wang, Wang, Gaskin, Hawk
specific: only passive
self-report
observational
2017
Rus, Tiemensma
general and specific measured but then aggregated
self-report
experimental
2017
Wei, Gao
general and specific
self-report
observational
2017
Chow, Wan
general
self-report
observational
2018
Worsley, Mansfield, Corcoran
general
self-report
observational
2018
Yuen et al.
specific
not clear if manipulation was checked
experimental
2018
Zeeni et al.
general
self-report
observational
2018
Hardy, Castonguay
general
self-report
observational
2018
Yang, Robinson
specific
self-report
observational
2018
Dhir, Yossatorn, Kaur, Chen
general
self-report
observational
2018
Weinstein
general
self-report
observational
2018
Booker, Kelly & Sacker
specific: chatting
self-report
observational: longitudinal
2019
Faelens et al.
specific: active vs. passive as well as public vs. private
self-report
observational
2019
Faelens et al.
specific: active vs. passive as well as public vs. private
self-report
observational
2019
Hall, Johnson, & Ross
general
self-report
experimental + ESM
2019
Han et al.
general
self-report
observational
2019
Orben, Dienlin, & Przybylski
general
self-report
observational
2019
Phu & Gow
general
self-report
observational
2019
Rui, Yu, Xu & Cui
specific (collected, but aggregated)
self-report
observational
2019
Twenge & Campbell
general
self-report
observational
2019
Xie & Karan
specific: 'directed communication' vs. 'broadcasting'
self-report
observational
2019
Heffer, Good, Daly, MacDonnell &Willoughby
general
self-report
observational: longitudinal
Note. Author lists may have been shortened to limit table size. The ‘Specificity’ column indicates whether the data gathered in the corresponding study was of a more general nature (e.g. frequencies, durations, platforms) or of a more specific nature (e.g. social media behaviours). The ‘Nature of Data’ column indicates whether the corresponding study used objective (e.g. retrieved from phone) or subjective (e.g. self-report) social media data. Lastly, the ‘Study Design’ column indicates whether the study included a manipulation (in which case it is noted as experimental), or not (in which case it is noted as observational).
Table 2: Data Log Overview of the Biggest Social Media Platforms
Facebook
Instagram
Twitter
Snapchat
· Account Status History
· Active Sessions
· Ads Clicked
· Address
· Ad Topics
· Alternate Name
· Apps
· Birthday Visibility
· Chat
· Check-ins
· Currency
· Current City
· Date of Birth
· Education
· Emails
· Events
· Facial Recognition Data
· Family
· Favorite Quotes
· Followers
· Friend Requests
· Friends
· Gender
· Groups
· Hidden from News Feed
· Hometown
· ID
· IP Addresses
· Locale
· Logins
· Logouts
· Matched Contacts
· Messages
· Name
· Name Changes
· Networks
· Pages You Admin
· Pending Friend Requests
· Phone Numbers
· Photos
· Photos Metadata
· Physical Tokens
· Pokes
· Political Views
· Posts by Others
· Recent Activities
· Registration Date
· Religious Views
· Removed Friends
· Screen Names
· Spoken Languages
· Status Updates
· Work
· Videos
· Comments
· Followers
· Messages
· Phone Contacts
· Photos
· Profile Information
· Saved Media
· Search History
· Settings
· Videos
· Who You Follow
· Your Likes
· A List Of Your Followers
· Ad Information That You’ve Seen Or Engaged With On Twitter Profile Information
· Interest And Demographic Information
· Lists
· Phone Contacts
· Who You Follow
· Your Direct Messages
· Your Media
· Your Moments
· Your Tweets
· Account History
· Bitmoji
· Bitmoji Kit
· Chat History
· Friends
· In-app Surveys
· Location History
· Login History and Account Information
· Our Story and Crowd-Sourced Content
· Purchase History
· Ranking
· Reported Content
· Search History
· Snap History
· Snapchat Support History
· Subscriptions
· Terms History
· User Profile
Table 3: Functionality Overview of the Biggest Social Media Platforms
Hello, i love this! I admit my righteousness and ambiguity on my actions with pure purpose for human longevity powered by energy of Gratefulness and Love. I need all the help i can get from human, cause everyone is a super hero in my eyes. I have 3 years of 200+ hours of recorded data on this process. I was wondering if you want to participate in a global movement initiated simultaneously across any/all platforms possible on a viral scale lets call it, “AWARENESS of Unawareness on Mental Well-being” i have gained the synchronicity of humanity through my pure intentions, life moments providing me to be a compiler of data beyond sciences answers, forever in configuration to create algorithms which entail the cross-analytical, meta-cognition, a symphony of synchronicity, with life harmony, crossed by Awareness of Awareness. I was looking for transcendence 3 months ago and gotten enlightened with visuals as DMT is what i can relate it too. Ive dedicated my end goal to end with speaking to the youth about the Internet and the values that need to be a common sense. As if an instructions for a foreign item that is nothing we thought we knew to experiencing. I even dug into adolescents and why those years from 12-16 are unlimited in potential compared to any other periods of life that align with everyone. It is beyond a passion for me, it is a vital to my well-being, a source of bliss, and the 1 soul that has the seed for the owner to nurture and harvest on their personal perception and clock. I have all my reasons with unlimited belief and my whole life as a track record to explain the severity it means to achieve this by December 15th, 2020. If you have any questions, comments, advice, peer analysis, or anything that you can assist in the process, i would more than be to compensate your time. I am homeless, no money, car, job, friends, family, or anything i can have a sentimental value till i have created enough noise to create a movement from snowball effect to an avalanche stopping when it is done. -Alexander Anh-Nam Tran -26 years young, Vietnamese-American, 1st Generation, Borderline of Z-generation [email protected] i am excited if you get this message on time, cause i only have a month to create this life changing moment to show the unlimited potential each person’s selves have stored behind dissonance, misconceptions, and just lost by false expectations.