There is a lively debate on the effects of social media use, shaped by self-reported measurements of social media use. However, self-reports have been shown to suffer from low accuracy compared to logged measures of social media use. Even though it is unclear how problematic that measurement error is for our inferences, many scholars call for the exclusive use of “objective” measures. But if measurement error is not systematic, self-reports will still be informative. In contrast, if there is systematic error, associations between social media use and other variables, including well-being, are likely biased. Here, we report an exploratory 5 day experience sampling study among 96 participants (435 observations) to understand factors that could relate to low accuracy. First, we asked what stable individual differences are related to low accuracy. Second, we explored what daily states relate to accuracy. Third, we explored whether accuracy relates to well-being. Although we did find evidence for a systematic tendency to overestimate social media use, neither individual differences nor daily states were related to that tendency. Accuracy was also unrelated to well-being. Our results suggest that blindly calling for objective measures foregoes a responsibility to understand measurement error in social media use first.
Keywords: social media use, measurement, personality, motivation, well-being
Funding: Grants from the Huo Family Foundation and Economic and Social Research Council (ES/T008709/1) supported Niklas Johannes and Andrew K. Przybylski. A grant from the European Research Council (851890; SOAR) supported Netta Weinstein. The funders had no role in study design, data analysis, decision to publish, or preparation of the article.
Conflicts of Interest: The authors declare no conflicts of interest.
Data Availability: The authors share all materials, data, and code on the Open Science Framework project for this article at https://doi.org/10.17605/ OSF.IO/7BYVT (Johannes, Nguyen, et al., 2021). The source code has been published on https://github.com/digital-wellbeing/smartphone-use. They documented all steps from raw data processing to final analysis on https://digital-wellbeing.github.io/smartphone-use/.
Correspondence concerning this article should be addressed to Niklas Johannes, Oxford Internet Institute, University of Oxford, 1 St Giles, Oxford, OX1 3JS, United Kingdom. Email: niklas.johannes@ oii.ox.ac.uk
When psychologists want to measure behavior, more often than not they ask people to estimate the frequency of those behaviors. Whereas self-reports are suitable for assessing subjective states and tendencies, such as well-being (Diener et al., 2018) or personality (Soto, 2021), self-reports of behavior are often inaccurate (for a critical discussion, see Chan, 2009). The question of behavior, accuracy, and measurement error has become increasingly relevant in the field of technology effects recently, particularly surrounding the effects of social media (Orben, 2020; Whitlock & Masur, 2019). This research topic affords researchers the luxury to objectively record social media use and compare the logs to self-reported (i.e., subjective) estimates of social media use. Therefore, work on social media allows researchers to examine how subjective and objective social media use as well as their discrepancy relate to other variables. Here, we examined these three types of reporting on social media use as follows: subjective (i.e., self-reported), objective (logged), and accuracy (amount of error between subjective and objective). To gain a more comprehensive understanding of social media estimates, we explored their relation to a range of stable individual differences and daily states that should be linked to social media use if personality and daily state indeed play a role for social media use.
Broadly speaking, inaccuracy in the form of measurement error can be exclusively random or a mix of random and systematic error. For example, people systematically overestimate their physical activity (Klesges et al., 1990) and underestimate their smoking (Connor Gorber et al., 2009)—both are examples of social desirability, one important factor in shaping inaccurate self-reports of behavior (for an overview of other factors, see Schwarz & Oyserman, 2001). As a counterexample, adolescents are inaccurate in recalling with whom they spent their breaks during school (van Woudenberg et al., 2020). But that inaccuracy is most likely random because recalling and aggregating all instances of a behavior from memory is difficult. When psychologists cannot measure behavior directly and resort to self-reports, both sources of error render our inferences less accurate. Although researchers can control for random error statistically, systematic measurement error poses a larger problem because it decreases validity of the measure and can bias relations with other variables (Niemi, 1993). For example, imagine a person whose low conscientiousness makes them overestimate their social media use, but also leads to lower well-being. For that person, higher social media use will be (spuriously) associated with lower well-being.
Unsurprisingly, people’s estimates of technology use are as problematic as other self-reports of behavior (Kaye et al., 2020). Whereas a handful of studies found underestimates (e.g., Ellis et al., 2019; Jones-Jang et al., 2020), other studies suggest that technology users overestimate how much time they spend with their devices and applications—be it the internet in general (Araujo et al., 2017; Scharkow, 2016), video games (Johannes, Vuorre, et al., 2021), SMS and calling (Boase & Ling, 2013; Vanden Abeele et al., 2013), smartphones (Sewall et al., 2020; Shaw et al., 2020), or social media (Verbeij et al., 2021). In a recent meta-analysis, Parry et al. (2020) found substantial variation between studies in conclusions regarding under- or overestimates. Across studies, respondents overestimated their technology use by a factor of 1.21, but that overestimate was not statistically different from a one-to-one ratio of subjective and objective use. More importantly, the authors found only moderate correlations between subjective and objective measures of technology use, below a threshold that would allow researchers to substitute subjective for objective measures. The authors concluded that subjective estimates are a poor measure of technology use. These shortcomings in measurement contribute to a literature that claims to observe negative effects of social media use on youths’ mental health, without acknowledging the limitations of relying on subjective estimates (Dienlin & Johannes, 2020; Orben, 2020).
The trend to overestimate social media use poses the question: Which factors play a role in systematic error? There is tentative evidence that stable user characteristics play a role in the accuracy of self-reported social media use. For instance, men tend to overestimate their use to a greater degree compared to women (Boase & Ling, 2013; Scharkow, 2016; Vanden Abeele et al., 2013), and older adults may overestimate their use relative to young ones (Vanden Abeele et al., 2013). These findings align with a psychology of media effects that has begun to move away from the universal effects of media use to a more nuanced understanding where personality traits play a key role in shaping media effects (Beyens et al., 2020; Valkenburg & Peter, 2013; Whitlock & Masur, 2019). Consequently, when we ask for whom media have an effect, it is equally important to ask who has low accuracy in their media use estimates. Understanding what individual differences, such as personality traits or general psychological need satisfaction, relate to over—or underestimates can inform a more rigorous research on media effects. For example, if those high in neuroticism experience social media use more intensely (Hisler et al., 2020), they might also overestimate their use. Such systematic error can bias the relations between social media estimates and mental health outcomes.
Because of a heavy reliance on subjective measures, there is little evidence regarding how personality traits correlate with social media use reporting accuracy. In a two-wave study focused on the Big Five, Andrews et al. (2020) found that only higher neuroticism predicted more self-reported social media use and vice versa. A meta-analysis examining the relation between the Big Five and various social media activities found that extroversion and openness were the most consistent predictors of a broad range of social media activities (e.g., posting status updates, interacting with other users), but most of these relations were small (Liu & Campbell, 2017). It is difficult to know, though, how accurate those findings are because of the low accuracy of self-reported social media use. Addressing that limitation, Prasad et al. (2018) reported that conscientiousness, neuroticism, and openness correlated with objectively measured social media use, but the authors did not measure subjective estimates to calculate accuracy. Similarly, several studies suggest that individual psychological need satisfaction is related to self-reported social media use (e.g., Lin, 2016; Sheldon et al., 2011), but none of those studies assessed accuracy. Taken together, previous research allows little insight into which individual differences relate to social media use and the ability to accurately estimate it. Here, we explore these questions by looking at individual differences in both personality traits (i.e., Big Five) and psychological need satisfaction (i.e., autonomy, competence, relatedness).
Besides stable individual differences influencing media effects, many scholars argue that we need to evaluate people’s states in everyday life, such as psychological need satisfaction and mood, to understand how and to what effect people use social media (Bayer et al., 2018; Meier & Reinecke, 2020; Valkenburg & Peter, 2013). The same logic should apply to the measurement of social media use. If, indeed, there are factors contributing to systematic bias in reporting social media use, people’s motivational and emotional states are a likely source of it. In general, emotions have shown to contribute to memory formation (e.g., Tyng et al., 2017). Positive affect has shown to contribute to working memory and, to a lesser degree, to short-term memory (Yang et al., 2013). Therefore, having a satisfying day that gratifies psychological needs might be associated with better memory of media use behaviors, which facilitates recall and thereby increases the accuracy of self-reported social media use. Conversely, boredom has shown to be related to time perception (Eastwood et al., 2012). Having a boring day might not only lead to higher social media use (Dora et al., 2020); it may also lead to perceiving time as slower, thereby decreasing accuracy. Such mechanisms would mean that people’s states relate to systematic error in the measurement of self-reported social media use. Therefore, we explored the associations between several daily states and accuracy.
Poor accuracy of social media estimates is consequential for understanding media effects because it lies at the heart of the debate around media use and well-being (Dienlin & Johannes, 2020; Orben, 2020). Overall, there seems to be a small negative between-person relation between social media use and well-being of approximately r = .10 (Dickson et al., 2019; Houghton et al., 2018; Orben et al., 2019; Orben & Przybylski, 2019; Schemer et al., 2020; Stavrova & Denissen, 2020; Thorisdottir et al., 2019; Vuorre et al., 2021), but no significant effects on well-being in studies that ask participants to reduce their daily social media use (Przybylski et al., 2021). Moreover, the vast majority of studies investigated self-reported estimates of media use, and so it is unclear whether that small relation is a consequence of measurement choice. The few studies measuring objective social media use yield mixed results. Some report small, negligible relations to well-being (Johannes et al., 2020; Katevas et al., 2018), some a mix of both null and negative relations (Rozgonjuk et al., 2018; Sewall et al., 2020), and others negative relations (Faelens et al., 2021). Overall, it is plausible that subjective estimates lead to an overestimation of the relation between social media use and well-being (Sewall & Parry, 2021; Shaw et al., 2020). That conclusion is far from definitive; recent work found that measurement made little difference for the relation (Jones-Jang et al., 2020).
Investigating accuracy in social media estimates is important to advance our knowledge on media effects. Just like we know little of who has low accuracy in their estimates, we know little about whether accuracy itself could indicate poor mental health. Only two studies have addressed this question. They show that discrepancies between objective and subjective social media use were positively related to depression, but also associated with higher life satisfaction (Sewall et al., 2020; Sewall & Parry, 2021). Because of the central role that accuracy plays in the debate surrounding effects of social media use on well-being, we explored whether accuracy relates to well-being and compare its relation to that of subjective and objective social media use.
We aimed to extend our understanding of accuracy in self-reported social media use. Importantly, we believe any factor that relates to bias in social media use estimates should be observable repeatedly in people’s live, on a daily level. Therefore, we employed a 5-day experience sampling design. Such a design will also reduce inaccuracies in self-reported media use because it is easier to recall behavior for a day compared to an estimate of typical media use (e.g., Whitlock & Masur, 2019). We had three research questions. First: Who has low accuracy in their social media use (i.e., person-level associations)? Understanding what stable individual differences (personality traits and need satisfaction) are related to social media use allows the field to test and advance theory on the role of individual differences in media effects. Second: What daily states relate to low accuracy (i.e., day-level associations)? Exploring that question advances our understanding of the conditions under which media effects might unfold. Third: Does accuracy relate to daily well-being (i.e., day-level associations)? Researchers currently debate whether relations between social media use and well-being are driven by how we measure social media use. Understanding how social media use and accuracy in its estimation relate to well-being can address that question. An experience sampling study is especially adequate to study well-being because media effects are generally small and transient (Orben & Przybylski, 2019). Consequently, researchers have argued that we should only observe effects on well-being on momentary well-being, not stable life satisfaction (Dienlin & Johannes, 2020).
To this end, we followed a sample of participants over time and collected data on (a) their stable individual differences, (b) motivational and emotional states, (c) daily well-being, and (d) subjective daily social media use. In addition, we recorded their objective social media use. The study was completely exploratory and we did not derive predictions from theory. Instead, we aimed to explore associations between prominent concepts in psychology and the field of media effects.
We share all materials, data, and code on the Open Science Framework project for this article at https://doi.org/10.17605/OSF.IO/7BYVT (Johannes, Nguyen, et al., 2021). We documented all steps from raw data processing to final analysis on https://digital-wellbeing.github.io/smartphone-use/ (from here on Online Supplemental Materials). Here, readers can also find the correlations between all variables.
We did not conduct a priori power analysis and aimed to recruit as many participants as we had resources for, which aligns with recent recommendations on feasibility analysis (Lakens, 2021). Therefore, we aimed to recruit 300 participants in a 1-month time window (mid-April to early May 2019). Our study was part of a larger project and consisted of three parts. First, we invited students who were 18 years or older to the lab to participate in a study that explores the relationships between individuals’ personality and their daily experiences. We created 300 time slots on the University of Rochester’s research participant pool hosted on the SONA recruitment platform. The slots were available to all psychology undergraduates that were registered on SONA. We obtained ethics approval from the University of Rochester (RSRB #3612; approval date April 9, 2019). Participants were told that they would first complete an initial survey, and would receive daily surveys from the following Monday to that Friday (i.e., the second part of the study). If participants had an iPhone with iOS 12.0 or higher, they could also sign up for an additional lab session after the 5 days of experience sampling ended (i.e., the third part of the study). Starting with 12.0, iOS has an inbuilt function to log screen time, pickups, and notifications called Screen Time. If participants were eligible and agreed to partake in the third part, we instructed them how to turn on the Screen Time functionality so that they would have logged social media use for the duration of the experience sampling portion of the study. Participants were explicitly informed that they would be asked to provide the amount of time they had spent on their phone and social media using this functionality.
Two-hundred ninety-two students participated in the first part, where they reported personality traits and demographic information.1 Of those, 275 participated in the second part. Here, participants received an email each evening asking them about the experiences and well-being on that day. The first survey took place on three Mondays: 15th, 22nd, and 29th of April 2019. Of those, 97 had an iPhone and agreed to participate in the third part of the study. For this part, participants came to the lab on the Saturday and Sunday after the last day of the experience sampling part (i.e., a Friday). Research assistants helped participants to go to the Screen Time function to record their social media use. The research assistants entered the data while the participants were present and the participants were explicitly told to stop the assistants anytime if they felt uncomfortable about the procedure. During this third part, the research assistants entered (a) total time, in hours and minutes, on social networking apps per day, Monday–Friday, the duration of the experience sampling part; that time window has been shown to provide an accurate estimate of participants’ typical phone use (Wilcockson et al., 2018), (b) their 10 most used social networking apps for the week, (c) total time and total number of notifications, (d) total time per app per day, and (e) how many total times participants used an app first after pickup for each day.
We took several measures to ensure high data quality. In the initial survey, two items evaluated inattention: “Choose [2/5] for this item (this item is to check for random responding”). Five participants failed both attention checks, but none of those five took part in the third part of the study. Therefore, we did not need to exclude their data because they were not included in the final sample as a result of not having screen time data. We also followed recommendations to identify meaningless and rushed responses by inspecting straightlining (Johnson, 2005; Leiner, 2013). For the initial survey, there were no instances of straightlining. However, there were several instances of straightlining for the experience sampling part. We identified those who straightlined on half or more of their total daily surveys. One of those also had screen time data, which is why we excluded that person from the analysis.
On the day level, we excluded one survey because the participant responded to the survey after noon the next day. Furthermore, we calculated accuracy as percentage error (see the Measures section). If someone estimates zero time on social media, we cannot obtain a meaningful percentage error. There were 11 daily surveys that had estimates of zero social media time. We set those times to missing, but retained all other data of these cases. Our final sample comprised 96 participants (M age = 20.5, SD age = 1.3, rangeage = 18–25; 66 women). The majority identified their ethnicity as Asian (40), followed by White (26), Black or African American (11), and Hispanic (10). For full details on the sample, see the Online Supplemental Materials. Among that final sample, the response rate to daily surveys was 91%, representing 435 daily surveys with valid data.
As we stated before, any factor that relates to bias in social media use estimates should be observable repeatedly in people’s lives. For such effects to be observable, they need to be moderate to large in size. As broad indication of our sensitivity, 96 participants (aggregating repeated measures) allowed us to detect such moderate to large effects. For an aggregate correlation between smartphone use and a personality trait, our study could detect effects of |ρ| = .28 at 80% power. However, repeated measures per participant can increase that sensitivity.
We assessed personality traits with the Big Five Inventory (John & Srivastava, 1999). The scales measure five personality traits that have shown to be robust and universal: extroversion, agreeableness, conscientiousness, neuroticism, and openness (Soto, 2021). Respondents rated 44 Likert-style items on a scale from 1 = strongly disagree to 5 = strongly agree. We aggregated the items for each scale to form a mean index per trait. See the distributions, mean values, standard deviations, and reliabilities in Figure 1.
We also assessed people’s level of need satisfaction with the Basic Psychological Need Satisfaction and Frustration Scale (Chen et al., 2015). The scale has 24 items assessing the fulfillment of 3 basic psychological needs with 8 items each: relatedness (feeling close and connected to others), autonomy (feeling volitional, choiceful, and self-congruent), and competence (feeling effective in meaningful tasks). These psychological needs drive effective self-regulation and well-being when they are fulfilled, and undermine both when thwarted (Ryan & Deci, 2000). Therefore, individual differences in psychological need satisfaction might relate to how much people use social media, how they perceive their use, and their accuracy in reporting their use. Respondents rated these items on a Likert-type scale from 1 = not true at all to 7 = very true. For each of the three basic psychological needs, we created mean indices. See Figure 1.
As well as relating to both self-regulation and well-being as an individual difference, psychological need satisfaction is also a strong predictor of well-being at the daily level (Reis et al., 2000). To assess daily states of psychological need satisfaction, participants filled out the 12-item Basic Psychological Needs and Frustrations Scale for diary measures (Mabbe et al., 2018). Respondents rated Likert-style items ranging from 1 = not at all true to 7 = very true. We aggregated the items per dimension (daily relatedness, autonomy, competence) to form mean indices. See Figure 2 for distributions, mean values, standard deviations, and reliability.
Just like daily motivational states, daily emotional states may relate to social media use and reporting. Therefore, we also assessed several emotions participants had throughout the day. Specifically, we asked how satisfied they were with their day, how boring their day was, how stressful their day was, and how enjoyable their day was. Participants rated these experiences on a Likert-type scale from 1 = not at all to 7 = very much. See Figure 2 for descriptive information.
To assess daily well-being, we asked participants to report how much they experienced each of low-arousal and high-arousal positive and negative emotions throughout the day on a 12-item version of the Positive and Negative Affect Schedule (Nguyen et al., 2018). For each dimension, participants indicated their mood on three Likert-type items, each ranging from 1 = not at all to 5 = very much. We reverse-coded negative emotions and aggregated all items into an overall mean index of well-being. See Figure 2.
On each day, we also asked participants to report how much time in total they spent on their phone using social media on that day. They filled in an estimate of hours and minutes into a text field.2 See Figure 3 for descriptive information of social media use.
In the third part of the study, research assistants extracted various metrics from the Screen Time app on participants’ phones. This was done to ensure accurate reporting. For objective social media use, researchers borrowed the phones while participants were present and recorded the apps that iOS classified as social networking in total for each day (i.e., Monday through Friday). For each day, the researchers filled in a text field for the number of hours and minutes. See Figure 3 for descriptive information. Objective and subjective social media use were correlated at r = .57. For a detailed comparison of objective and subjective use per participant, see https://digital-wellbeing.github.io/smartphone-use/descriptives-and-visualizations.html#fig:dumbbell-social-media.
We calculated accuracy as percentage error. Most research has relied either on difference scores (Boase & Ling, 2013) or an absolute difference score (Sewall et al., 2020). However, neither of these measures is intuitive to interpret: If someone uses social media 20 min more than they estimated, that person can be highly accurate if the true number is 400 min (in which case 20 min is a small proportion of time spent; 5% error), or highly inaccurate if the true number is 5 min (in which case 20 min is a large proportion of time spent; 400% error). Percentage error can put these differences into perspective (Vanden Abeele et al., 2013). It is calculated as .
Therefore, percentage error gives a more sensible estimate of accuracy taking into account the relative difference in estimated and objective social media use. Furthermore, rather than taking the absolute difference, it allows both negative and positive accuracy, which represents underestimates and overestimates. However, percentage error is meaningless if someone estimates zero social media use. The error compared to nothing cannot be interpreted and the formula will always return −100%. We therefore set 11 data points who estimated zero social media use to missing (see also the Participants and Procedure section). See Figure 3 for descriptive information on accuracy.
We analyzed three blocks of models in R (R Core Team, 2020). There were three models per block, for a total of nine models. The first block asked about the relation between person-level variables (i.e., Big Five and psychological need satisfaction) and subjective social media use, objective social media use, and accuracy. The second block asked about the relation between day-level variables (i.e., motivational and emotional states) and subjective social media use, objective social media use, and accuracy. The third block asked about the relation between day-level social media variables (i.e., subjective social media use, objective social media use, and accuracy) and well-being. For each model, we estimated a Bayesian multilevel level model with the R package brms (Bürkner, 2017). For slopes and intercepts, we employed weakly informative, regularizing priors; for all other parameters we relied on the brms default priors. For example, for the relation between (standardized) accuracy and well-being we relied on previous studies who found a small negative relation (Sewall et al., 2020); we therefore assumed that slopes (a) would be normally distributed, (b) centered on a small negative mean of −.2 Likert-points, and (c) indicate generally small to moderate effects with 95% of slopes between −1.0 and 0.60 on the 5-point Likert-scale. For full details on the priors, see the online Supplemental Materials (https://digital-wellbeing.github.io/smartphone-use/analysis.html).
The first block predicted the social media variables and accuracy with person-level variables. We did not choose a normal distribution for the social media variables because we know that minutes cannot be less than zero, the scale is continuous, and the variance for time with an activity often increases with the mean. Therefore, we assumed a Gamma distribution with a log link as data-generating process. We grand-mean centered all person-level predictors and included random intercepts per participant and per day. For accuracy, we assumed a normally distributed data-generating process, but wanted to allow fatter tails to account for occasional large inaccuracy in both directions. Therefore, we assumed a student-t outcome distribution.
The second block predicted the social media variables and accuracy with day-level variables. Again, we assumed a Gamma distribution for the social media variables and a student-t distribution for accuracy. To separate between-person and within-person processes, we followed recent recommendations and for each predictor entered the person mean and the deviation from that person mean (Hamaker & Muthén, 2020). In addition to including random intercepts for person and day, we followed best practices in multilevel modeling and included random slopes for within-person effects (Barr et al., 2013), which gives us a better claim to generalize our findings (Yarkoni, 2019).
The third block predicted well-being with the social media variables and accuracy. For well-being, we assumed a data-generating process that results in a normally distributed outcome. We again separated between-person and within-person processes for both the social media variables and accuracy. We also included random slopes for within-person predictors.
We inspected model fit with posterior predictive checks and leave-one-out cross-validation (McElreath, 2020). Readers can find full model diagnostics in the online Supplemental Materials (https://digital-wellbeing.github.io/smartphone-use/analysis.html).
We tested our three research questions with three models each and, conditional on our model assumptions, the data were not in line with the idea that there are true effects linking individual differences, motivational and emotional states, accuracy, and well-being. The first research question concerned individual differences on the person level, namely, how the Big Five and psychological need satisfaction relate to daily objective and subjective social media use and accuracy. Conditional on our model assumptions, the data were incompatible with large true effects for personality-level traits and psychological needs. Figure 4 shows that none of the 95% posterior distributions excluded 0 (in the case of accuracy) or 1 (in the case of odds for social media use). Of all predictors, only the posterior distribution of neuroticism almost excluded the null effect. Scoring one point higher than average on the neuroticism trait was associated with a 22% underestimate of social media use. Similarly, as neuroticism increases one point above average, objective social media use increases by a factor of 1.12. Again, the 95% of estimates that are most compatible with our data include a null effect, rendering those associations less convincing. For full numerical details of the coefficients see the online Supplemental Materials (https://digital-wellbeing.github.io/smartphone-use/synthesis.html#numerical-model-estimates).
Our second research question concerned variables on the day level, namely, how motivational and emotional states relate to subjective and objective social media use and accuracy. We find a similar pattern for day-level predictors as for the first research question. Figure 5 shows that neither differences between people (between-person effect) nor people’s deviations from their typical states (within-person effect) are associated with meaningful changes in accuracy or subjective or objective social media use. Only deviations from people’s typical state of satisfaction and boredom might be related to accuracy. If a person scores one point higher on satisfaction than they typically do, they underestimate their media use by 6% on average. By contrast, reporting one point higher boredom than a person’s average boredom is associated with a slight overestimate (3%) of social media use. However, the 95% credible intervals once again contained zero.
Our third research question concerned well-being on the day level, namely, how subjective and objective social media use, as well as accuracy, relate to daily well-being. We find that neither subjective social media use nor objective social media use nor accuracy is meaningfully associated with daily well-being. Figure 6 shows that plausible associations between a person who estimates to spend one hr more on social media than another person and well-being are small and contain zero. The same goes for estimating to spend 1 hr more on social media than a person typically does. These between-person and within-person patterns are similar for objective social media use. As for accuracy: Overestimating one’s typical social media use by one standard deviation on accuracy is associated with a 0.03 decrease on the 5-point Likert-scale for well-being—a small association at best and one that might just as well be small and positive given the width of the Credible Interval.
Finally, we wanted to explore the relation between subjective and objective social media use in more detail. If we treat objective social media use as the true value predicting self-reported social media use, such a model can give us an idea of the amount of bias in self-reported social media use. If there truly were only random error, the intercept of such a model would be close to zero. In other words, at 0 min of objective social media use, does the model estimate subjective social media use to also be at zero with some variation (i.e., random measurement error) or different from zero (i.e., systematic measurement error)? We predicted social media use with objective social media use in another multilevel model, with random intercepts for participant and day and a random slope per participant. This model indeed suggested that there was systematic bias. The intercept was estimated to be far away from zero, estimate = 71, 95% CrI = [50, 93], which indicates evidence that the overestimate of social media time is not random, but systematic. The full model is in the Online Supplemental Materials. Therefore, we found evidence for systematic measurement error in social media estimates, but did not identify potential factors that could explain that systematic error.
Most research on social media effects asks participants to provide an estimate of their use. These estimates suffer from measurement error, with only overlap between self-reports and tracked use (Parry et al., 2020). But how problematic is this measurement error? If error is random, self-reports might still be informative, if noisy. By contrast, if in addition to random error there is systematic error in self-reports, the associations between self-reported social media use and other variables are likely biased (Niemi, 1993). Previous research strongly suggests that error is indeed systematic because people generally overestimate their social media use (e.g., Araujo et al., 2017; Scharkow, 2016; Vanden Abeele et al., 2013). As a result, many researchers have called for the field to adopt more objective measures (Dienlin & Johannes, 2020; Orben, 2020). Here, we explored several candidates that could explain low accuracy in self-reports of social media use. We studied whether stable individual differences (personality traits and psychological need satisfaction) and transient daily (motivational and emotional) states can predict lower accuracy in reporting, and whether accuracy relates to well-being. We indeed found evidence for a tendency to systematically overestimate one’s social media use. However, neither person-level nor day-level variables were meaningfully related to that tendency; and low accuracy did not correlate to daily well-being to a meaningful degree.
First, we asked whether stable individual differences were related to social media use and accuracy. Such relations would be in line with a call for taking individual differences into account when we study media effects (Beyens et al., 2020; Valkenburg & Peter, 2013). Our results do not support that view. Neither the Big Five traits nor individual differences in psychological need satisfaction showed a meaningful association with subjective use, objective use, or accuracy. The results mostly line up with previous research that showed little relevance for personality traits when explaining social media use (Andrews et al., 2020; Liu & Campbell, 2017). That said, those traits in our analysis that were most predictive of more social media use (even if they are compatible with no effect) were neuroticism and extroversion. Neuroticism was the only reliable predictor of subjective social media time in a previous large-scale study (Andrews et al., 2020); extroversion was a robust predictor of various social media activities in a meta-analysis (Liu & Campbell, 2017). However, previous work rarely collected data on the day level. Therefore, the most prominent traits that might relate to daily social media use are extroversion and neuroticism, but these associations are likely small. As for accuracy: Only neuroticism and openness came close to being incompatible with a null effect, predicting lower accuracy (around 20% lower for each increase in the traits). Without an objective benchmark of accuracy, though, it is difficult to judge whether such decreases in accuracy are meaningful. Overall, our study finds little evidence that personality characteristics relate to systematic error in the measurement of social media use.
Second, we asked what daily motivational and emotional states are associated with accuracy. Specifically, we followed the proposition that associations with media use might be transient and situational (Bayer et al., 2018; Meier & Reinecke, 2020) and explored how several of these transient states correlate with social media use. Just like with stable individual differences, we found little evidence that daily states are related to social media use and accuracy. Neither on the between-person level nor on the within-person levels were there large associations between any of the daily states and social media use or accuracy. In fact, most associations were extremely close to no effect, which makes it unlikely that the true associations between these states and social media use are large. Our results run counter to the idea that daily experiences shape social media use or the accuracy with which people report that use. Whether people experience psychological need satisfaction or have an enjoyable, boring, or satisfying day seems uncorrelated to the systematic error we observed in social media use estimates.
Third, we asked whether accuracy relates to well-being. There is a lively debate in the literature whether social media use has a negative impact on well-being (e.g., Orben & Przybylski, 2019). But most studies rely on self-reported media use and it is unclear whether any association between media use and well-being—in itself questionable (Przybylski et al., 2021)—becomes weaker or even disappears once we use objective measures (Jones-Jang et al., 2020; Sewall & Parry, 2021; Shaw et al., 2020). Moreover, some initial evidence suggests that those with low accuracy in their self-reports also feel worse (Sewall et al., 2020; Sewall & Parry, 2021).
Our findings contribute to the debate in two ways. First, it made little difference for the association with daily well-being whether we measured social media use as self-report or objectively. Both estimates were remarkably similar and, more importantly, most compatible with the lack of an effect. In other words, our results suggest that the way we measure social media use does not affect its relation to well-being—because social media use seems unrelated to well-being no matter how you measure it. Second, accuracy was not related to well-being to a meaningful degree. In contrast to previous studies where accuracy was negatively related to well-being (Sewall et al., 2020; Sewall & Parry, 2021), we found that accuracy was not indicative of lower well-being. If daily social media use had a large short-term effect on how we feel on a day, we should have been able to pick up that effect. Not finding a negative association, in our opinion, constitutes strong evidence against the negative effect of social media time.
Overall, we do not find that the person-level or day-level variables we investigated are related to measurement error of social media use to a meaningful degree. However, that lack of a relation does not mean that error is random. Our analysis suggests that the overestimate of social media use was robust and systematic. This finding contributes to the literature which has yet to show whether people generally report more or less technology use than they truly engage in (Parry et al., 2020). Moreover, our findings add nuance to that literature, showing that overestimates can occur consistently in measurements across several days, not just in cross-sectional studies. Consequently, there appears to be systematic error in technology use estimates, but individual differences and motivational and emotional states (at least the ones on which we focused) cannot explain that bias. Moreover, our study raises doubts whether such bias plays a role in media effects on well-being. Even systematically overestimated social media use was unrelated to daily well-being and its estimate close to identical to that of objective social media use. Therefore, our findings suggests that we can learn from studies that use self-reports to investigate the relation between social media use and well-being—even if those self-reports will be inaccurate to a degree.
We must qualify that conclusion. We measured social media use on a day level, and activities in that short unit of time may be more concrete and salient in recollections. For example, imagine being asked to report on your sugar intake on a given day. It would be quite easy to recall the nature and amounts of foods that you ate. On the other hand, quantifying sugar intake across a period of 1 month becomes more elusive. Most research on social media use and well-being is either cross-sectional, asking about typical use, or longitudinal, asking multiple times about a longer reference period. We do not know whether the lack of a difference in measurement is consequential for well-being for these time frames, simply because there are no data on that question.
Naturally, our study has several limitations. First, we might well have been underpowered to detect small effects. Effect sizes in media effects research are typically small (Rains et al., 2018). The literature shows that most effects of media use on well-being are close to null on the within-person level, and small on the between-person level (Dienlin & Johannes, 2020; Orben, 2020). Even though we had multiple measures per person, five measures might not be enough to reliably detect such effects. However, effects on well-being need to be moderate to large for people to subjectively feel them (Anvari & Lakens, 2019; Norman et al., 2003), which our study should have been able to detect. As for individual differences, we can only reject the claim that the effect of individual differences on media use or accuracy is large.
Second, we want to emphasize that we cannot and did not make causal claims; we conducted an exploratory study and merely looked at associations between variables. Third, studies comparing subjective with objective use often alert users to their media use. Ours was no exception: Participants who came back to the lab to report their objective social media use might have felt prompted to pay attention to the time they spent on social media during the week. Increased attention to one’s own phone use might explain the relatively high correlation between subjective and objective social media use we found. Then again, our participants were informed explicitly that they would be asked to provide the amount of time they had used their phone and social media, but they were not instructed to be accurate when responding to daily surveys to report on their subjective use. If anything, if participants looked up their screen time before each survey, they would have been more accurate in their estimates; therefore, the accuracy (and the overestimate) in our study would be an underestimate.
Fourth, to obtain objective measures of social media, we relied on the built-in classification of iOS. However, iOS might classify apps as social media that participants do not consider a social media app. Similarly, participants might have reported subjective estimates of their social media use that did not occur on a phone. Therefore, the discrepancy in what participants consider a social media app and what iOS considers a social media app might have contributed to a higher discrepancy between the two measures. However, if iOS has a broader definition of social media than participants, we should have observed an underestimate in use time. Alternatively, the overestimate we observed might be more pronounced if the objective measure included less apps than participants felt were appropriate to consider in their responses. Finally, participants reported their states for the entire day at the end of the day. Asking participants to aggregate all instances of a state across the day might have introduced more stability to the state measures and not have captured moment-to-moment variations in states. As a consequence, our measures can be considered closer to daily diary measurement than to experience sampling in the narrow sense. That said, the method was adequate to answer our research questions on the day level. We need more work that looks at how variations in moment-to-moment smartphone use relate to moment-to-moment variations in states.
The field of media effects research has been calling for more “objective” measures of media use, largely because self-reports are now known to be inaccurate. But who has low accuracy in their social media use estimates? What motivational and emotional states are associated with low accuracy? And does accuracy correlate with well-being? We indeed find evidence that self-reports suffer from systematic measurement error: people overestimate their use. But we do not find evidence that individual differences or daily states meaningfully relate to that error. Type of measurement and accuracy also do not seem to matter when looking at the relation between social media use and well-being. Our results suggest that researchers cannot blindly dismiss the results of studies that rely on self-reported media use when studying well-being. We might still learn from them. Instead, we need to understand the source of systematic bias in these self-reports.