RUTHFULLY YOURS

Flaws in a Recent Lancet Study on Phone Use in Schools Five problems that call into question the authors’ conclusion that phone restrictions don’t improve mental health or academic performance Jon Haidt, Zach Rausch, and Alec McClean

Posted By Ruth King on February 14th, 2025

https://www.afterbabel.com/p/lancet-study-flaws?utm_source=post-email-title&publication_

In this post, we show why the recent Lancet study’s assertions that restrictive phone policies in schools yield no benefits are unfounded. For example, the phone-policies in the ‘permissive’ vs ‘restrictive’ schools did not differ very much, their measure of academic performance was crude, and their measures of screen time were unreliable.

A recent study published in The Lancet (Goodyear et al., 2025) has generated news headlines suggesting that restricting phone use in schools has no effect on the wellbeing or academic performance of students. This contradicts several previous studies that did find such benefits.

In this post, we lay out several flaws in the design and interpretation of the Lancet study, and several oddities in the data that we believe render its “no benefit” conclusion unjustified.

The authors of the study claimed to have

[E]valuated the impact of school phone policies by comparing outcomes in adolescents who attended schools that restrict and permit phone use.

The word “impact” implies the ability to discern causality. The authors then assert that,

[T]here is no evidence that restrictive school policies are associated with overall phone and social media use or better mental wellbeing in adolescents

and conclude that

[T]here is no evidence to support that restrictive school phone policies, in their current forms, have a beneficial effect on adolescents’ mental health and wellbeing or related outcomes, indicating that the intentions of these policies to improve adolescent health, wellbeing, and educational engagement are not realised.

The authors note that they do find substantial associations between time spent using phones or social media and worsened mental health and wellbeing, physical activity and sleep, and attainment and disruptive behavior:

[T]he negative associations found between increasing time spent on phones/social media and worsened mental health and wellbeing do provide evidence on the need to address phone and social media use in adolescents, and school policies should be developed as part of a more holistic approach.

Within each school, the heavy users of smartphones and social media are doing worse than light users, across multiple important outcomes including mental health and educational attainment. However, the fact that the two groups of schools did not differ on average leads the authors to conclude that while there are consistent associations with harm, phone-free policies alone don’t reduce those harms.

The study’s strong claims (“there is no evidence”) and assertions about the supposedly minimal “impact” of restrictive policies are likely to be interpreted by many as evidence against restrictive policies in schools. For example, here are two recent headlines about the study’s findings:

The purpose of our preliminary review of the Lancet study is to point out several key flaws and shortcomings in the study’s design and methodology. Given the large scope and complexity of the study, this review is not comprehensive. Nonetheless, this preliminary review should caution researchers, educators, and legislators to avoid taking this study as evidence that phone-free school policies do not improve academic outcomes or student mental health.

Share

What the Lancet Study Did

The authors recruited 20 “restrictive” and 10 “permissive” schools. Within each school, they surveyed one “year eight” (ages 12-13) and one “year 10” (ages 14-15) class. The student survey included a wellbeing evaluation on the The Warwick-Edinburgh Mental Wellbeing Scales (WEMWBS) as well as self-reported estimates of time spent using phones and social media during the school day, and outside of school. The study also surveyed teachers, asking them to classify students academically and behaviorally (see Supplementary Appendix for details and all the outcomes).

The study reported no differences between the two groups of schools on either academic outcomes (as reported by teachers) or on mental health (as reported by students). That was their primary finding. Does that finding suggest that phone-free policies have no impact? We see 5 problems in the study that should make people wary of drawing that conclusion.

Problem #1: The Permissive Schools Were Not Very Permissive; Most of the Restrictive Schools Were Not Phone-Free

In Jon’s 2023 Atlantic article (ungated copy here) where he first made the case for phone-free schools, he laid out six different levels of phone restrictions along a continuum from 0 (no policy at all) through phone-free (levels 4 and 5).

Level 0: No restrictions on phones

Level 1: Students can take their phones out during class, but only to use them for class purposes.

Level 2: Students can hold onto their phones, but are not supposed to take them out of their pockets or backpacks at all during class time.

Level 3: Phone caddies in classrooms: Students put their phones into a wall pocket or storage unit at the start of each class, and then pick them up at the end of that class.

Level 4: Lockable pouches. Students are required to put their phone into their own personal pouch when they arrive at school, which is then locked with a magnetic pin. Students keep the pouch with them but cannot unlock it until the end of the school day.

Level 5: Phone lockers. Students lock their phones into a secure unit with many small compartments when they arrive at school. They keep their key and get access to the phone lockers again only when they leave school. (We note that a low-cost “envelope policy” in which students deposit their phone in a large envelope with their name on it, which is kept in a bin near the homeroom teacher’s desk, would count as Level 5.)

The headlines about the Lancet article might lead a reader to assume that the study contrasted schools with the lowest levels of policy (0 or 1), which are quite permissive, with schools with the highest levels of policy (4 and 5), which are phone-free. But what sorts of school policies were actually contrasted in the Lancet article?

The study first examined school phone policies of 1341 secondary schools within a certain region of England in order to dichotomize the schools into a permissive and a restrictive category. (This categorization was drawn from an earlier paper, by Randhawa et al., 2024). Note that only 7.2% (96) of the schools were found to have permissive phone policies. Out of these 96 “permissive” schools, they recruited 10 to be part of the study.

Note how permissive school policies were defined: “[P]hones were permitted to be used at any time or at certain times (e.g., breaks/lunch) and/or in certain zones (e.g., outside).” Of the ten permissive schools, nine had classroom ban policies, and only one allowed phone use at any time throughout the school day. In other words, permissive schools were typically schools with classroom phone bans already in place. Nine of them were at Level 2 or 3, and only one was at Level 1.

Conversely, note how restrictive schools were defined: “Required phones to be inaccessible to pupils.” That sounds good, but only four of the twenty restrictive schools had policies that made phones truly inaccessible. One used locked pouches, one used phone lockers, one required that students hand in their phones to the front office, and one did not allow phones on the school premises. The other 16 schools seem to have had a “backpack policy” (phone use was not allowed during the school day, but students could keep them in their backpacks, and were supposed to power them down.)

A backpack policy would fall somewhere between levels 3 and 4. It can work if very strictly enforced, but leaving the phone accessible to students often means that many students sneak a quick peek throughout the day, sometimes in the bathroom. In other words, only 4 of the schools met our definition of a “phone-free school,” which is levels 4 and 5. Just knowing that one could be checking texts and social media posts, and knowing that some other students are texting and posting, would reduce the benefits of making phones fully inaccessible.

In summary, this study compared school policies we would define as Levels 2 or 3 against school policies we would define as falling between Levels 3 and 4. Understanding these levels is important, because it’s important to know what sort of policies are actually being compared in the study. Readers and journalists may have assumed from the headline that the study compared a set of more permissive policies (i.e., where students could use their phones anytime) to a set of more restrictive policies (i.e., schools that went truly phone-free, with phones inaccessible) than the study actually examined. In reality, the study was primarily a comparison of schools with a classroom ban versus schools that let students keep their phones with them at all times, in their backpacks. We cannot know how many of these students truly turned off their phones and kept them in their backpacks all day long.1

We note that in the supplementary information the authors did re-run some of the analyses, limiting the restrictive schools to the four truly phone-free schools, and they report that it did not change the results. However, these analyses were still only comparing classroom bans (not truly permissive policies) to phone-free schools. These analyses are also relying on a very small sample of restrictive schools, which limits their ability to detect any differences and to be confident that these four schools are representative of the full population of truly phone-free schools.

Problem #2: Unsubstantiated Causality

Although the authors claimed that their study was “designed to evaluate the impact of school phone policies,” they did not explain how their methodology could discern causality, and, in fact, recognized the study’s inability to do so. They state: “A limitation is that the study design is cross sectional which makes it difficult to draw conclusions about causality and reverse causality cannot be ruled out.”

Even with this acknowledged limitation, the authors, at times, conflated correlation with causality, whether indirectly, by switching between assertions about impact and associations in the Abstract, or directly, such as when stating:

In addition, there is no evidence to support that restrictive school phone policies, in their current forms, have a beneficial effect on adolescents’ mental health and wellbeing or related outcomes, indicating that the intentions of these policies to improve adolescent health, wellbeing, and educational engagement are not realised.

That statement makes sense only if the study was able to discern causation yet found no evidence for it. The data in this study was correlational—they did not conduct a study designed to estimate a causal effect. For instance, they might have collected data before and after schools implemented phone policies (as was done here in a study that did find improvements in academic performance), or they might have randomized schools to implement different phone policies (which would be ideal, but is very hard to do). Therefore, as the study acknowledges, systematic differences between permissive and restrictive schools (such as by social class, race, or gender ratios) could cause differences in academic outcomes that are unrelated to their phone policies. The authors make a reasonable effort to control for such systematic differences, but cannot rule out their presence.

In particular, an unexamined systematic difference between the two types of schools was the students’ phone use outside of school. Although students at restrictive schools used their phones less in school, they used them more outside of school, so that students in permissive and restrictive schools used their phones, on average, about the same amount (See Table 3). The analysis does not account for this systematic difference in out-of-school phone use, but it could be the most crucial confounder in their data. Heavy phone use outside of school could harm mental health while also motivating schools to adopt phone restrictions. The fact that this study does not examine this relationship in detail also harms its ability to infer causality.

Leave a comment

Problem #3: Academic Results Hinge on a Single Binary Rating

Readers might assume that the study compared objective academic outcomes across the two groups of schools. Which ones have better academic outcomes? But even though such school-wide data is available, the study did not use it. Instead, it seems the conclusions about academic effects are dependent entirely on two single-item responses by form teachers (form teachers are the UK version of a U.S. “homeroom” teacher). The form teachers were instructed to access the ratings made by maths teachers and english teachers, but these ratings were not grades or test scores, they were evaluations made by the teachers as to whether each student was working at, above, or below their target grade. The researchers then reduced these trinary ratings down to a binary rating: below target, or not below target (it is not clear what is meant by “target”). This binary variable contains very little information about each student’s performance. It would take a very big intervention to push a sizable group of students from one category to the other, which is what would be needed to produce a statistically significant result.

Confusingly, the authors suggest that this is an objective evaluation when they use the word “data” in the following statement: “The form teacher for each class was recruited and asked to provide data on pupil attainment and behaviour.”

The subjective nature of the classification is explained only in the Supplement (see Supplementary Table 1: Outcome measures and handling of missing data), but the study text gives the impression—and subsequent news coverage does too—that academic results were based on objective or semi-objective measures like test scores or grade point average (GPA).

Problem #4: The Failure of Efforts to Obtain Objective Phone Use Data

The students were asked not only to estimate the time they spent on phone and social media, but also to report the phone and social media times calculated by the ScreenTime app on iPhones or the Digital Wellbeing app on Android phones. But this effort to get more objective data from the phone itself failed, and the authors ultimately abandoned any use of the app data:

“Data were collected from adolescent phones on screen time and social media use; however, we were unable to include these measures in our analyses, due to concerns related to the accuracy of adolescents self-reporting phone and social media data from apps, as well as a high proportion of implausible data.”

Elsewhere the authors note: “There was a high proportion of missing data due to input errors and the way the phone apps may have been miscounting social time, resulting in nearly a third of participants reporting higher social media times than phone times.”

If nearly a third of students reported higher social media time than phone time then something is clearly going wrong with the procedure and the measure is unreliable. We are left with students’ own estimates of their screen time and social media time, which has its own set of problems (see problem 5).2

Problem #5: The Screen and Social Media Estimates are Odd

It turns out that the self-report estimation of time spent on phones and on social media also proved difficult, although this is unmentioned in the study text itself – one can find this in the Supplementary Appendix.

In the Data Cleaning Notes of the Supplement, the authors explain:

When social media time was greater than Smartphone Time, instead of excluding it like we did for the phone reported data, we set the Smartphone Time to equal the social media time as previous research has found that self-reported social media time tended to be more accurate than self-reported Smartphone Time. This affected 12.39% of participant’s self-reported school day Smartphone Time data, 9.21% of participant’s self-reported weekend Smartphone Time data, and 13.85% of participant’s in school Smartphone Time data.

In other words, 10% to 15% of students estimated their social media time to be higher than their phone time. Furthermore, the solution to this problem by the study authors—to set phone time to match social media time—appears questionable.3

In short, both the phone use and social media use estimates data appear unreliable, whether those estimates are based on readings of apps or on simple self-report by students.4

Share

Conclusion

Thousands of schools have gone phone-free (bell-to-bell, levels 4 and 5) in the last year or two, with many beginning just this past September. Educators, administrators, and legislators around the world want to know: How is it going? What happens in a school after it changes its policy? The reports we can find nearly always say that there were numerous behavioral benefits, including declines in cyberbullying and tardiness, improved attentiveness in class, and more laughter in the hallways and lunchrooms. And the available natural experiments that zoom in on the effects of restrictive bans typically find benefits to student learning, behavior, and well-being.

The Lancet study is innovative in that it collected mental health data directly from students. Most prior studies have focused on the changes in academic and behavioral outcomes and have not had data on mental health. But the Lancet study does not let us conclude anything about the effects of changing phone policies on student mental health or academic performance. The Lancet study simply tells us that a set of students in each of two small sets of schools—which differed in many ways, including in their phone policies—did not also differ in their academic performance (assessed by a single teacher rating) or in their mental wellbeing.

Clearly more research is needed to understand the effects of each of the different kinds of phone policies. With so many schools going truly phone-free (levels 4 and 5), we hope that researchers will work with schools to obtain key measurements before the policy is changed, and again in the months and years after the policy is changed. It would be especially helpful to study the change in schools that have used the same measures for several years to track mental health, as well as academic performance and behavioral issues (such as absenteeism, tardiness, fights, vandalism, and cyberbullying).

Ideally a group of schools could be identified in which some schools will change their policies at the start of the next school year, while other schools maintain a permissive policy. That would provide a control or comparison group (although not one selected by random assignment), which would allow a “difference in difference” study that compares the changes in the schools that changed policies to the changes in schools that did not change policies.”

While the headlines inspired by this article may discourage school administrators and legislators from adopting phone-free policies, we believe the following headlines are more illustrative of what happens in schools that go phone-free:

Phone free schools are much better for teachers, and they are much more fun for students. As one school principal explained,

The remarkable thing for me is many of the students, in about week two or week three [of the new policy], came up to me and said ‘Thank you for doing this Mr. Kilpatrick’.

Comments are closed.