ICTs opportunities and risks: Effectiveness of a nationwide intervention

Introduction

The use of digital technologies has become a core part of people’s everyday lives, particularly for youths (Ang et al., 2012; Díaz-Vicario et al., 2019; Ortiz Sobrino et al., 2019; Salehi et al., 2014). Through Internet and Communication Technologies (ICTs), adolescents can freely express themselves, keep in touch with friends, and search for information (Bulfin et al., 2016; De Mello et al., 2019; Lašek et al., 2016). The positive aspect of ICTs can lead young people to spend a lot of time online, up to an excessive use of electronic devices (Katz et al., 2014; Smahel et al., 2012, 2020).

In fact, the online world offers opportunities but also hides a multiplicity of risks that young people may run into. It is therefore essential to make them aware of these risks and how to use digital technologies in a positive way. This aspect became even more crucial in the period characterized by the COVID-19 pandemic. Along with a general increase in the use of ICTs, students were given the possibility of continuing their studies remotely, thus allowing them to maintain social relationships with peers and teachers (Öçal et al., 2021; Pokhrel & Chhetri, 2021). However, the increased use of ICTs has also been linked to an increase in exposure to online risks such as internet addiction and cyberbullying (Lin, 2020; Mkhize & Gopal, 2021). This is the context in which understanding the impact of interventions aimed at promoting a positive use of ICTs, and raising awareness of online risks, appears to be even more relevant.

In recent years, many programs have been developed to deal with specific risks (e.g., cyberbullying; Gaffney et al., 2019), to improve the digital skills of teachers and students (Kralj, 2014) and to promote innovative teaching practices through the use of digital technologies (Pombo et al., 2017). Although an ecological, systematic approach has been found to be more effective, the programs quite often target a single actor (e.g., just youths or teachers). In the European context, Safer Internet Centre project (SIC) have been set up in each country thanks to grants funded by the European Commission. They promote initiatives on a national level to make the Internet a better and more trustable place for youths on the basis of three core components: the Awareness Centre, Hotlines to report harmful online contents (e.g., child pornography), and a Helpline to take care of children who report online abuse or issues. Each country decides the specific initiatives to carry out within the Awareness Center. Although SIC project have been settled up since 2012 no scientific evaluation has been published on the effectiveness of these projects.

In the evidence-based literature, much attention is given to the evaluation of the stage of scaling-up in real world condition and the evaluation of the effects of the dosage of the intervention (Gottfredson et al., 2015). Although these stages of evaluation are quite hard to achieve, these sorts of efforts are encouraged in order to have effective interventions able to be transferred in different contexts.

The objective of the present study is to evaluate the effectiveness of the nationwide awareness activities led by the SIC Italy project, taking into consideration the moderator role of the dosage of the intervention in the real-world condition of implementation.

Opportunities vs. Risks in the Online Context

There are many contexts in which digital technologies may offer opportunities. First, in the educational setting. Since early childhood, the use of digital devices in an appropriate way, and together with other traditional tools, can support learning processes (Bajovic, 2018). The development of students’ digital skills since primary school is crucial (Pöntinen & Räty-Záborszky, 2020) along with keeping the schools constantly updated on the technological changes that are taking place in the digital age (Salehi et al., 2014). Communication and sociality can profit as well. The impact of ICTs on social interactions is evident: today it is possible to cut distances between people and keep in contact without having to meet personally, changing the structure of relationships (Betts & Spenser, 2017; Betts et al., 2017; Ortiz Sobrino et al., 2019).

However, people surfing online are unaware of the risks they might face (Letica, 2019). There are many risks associated with youths’ misuse of the internet. For instance, violation of privacy, cyberbullying, sexting in its coercive and non-consensual forms, online grooming, and sexual abuse as its possible consequence, and more (Wurtele & Kenny, 2016). A recent report of EU Kids Online (Smahel et al., 2020) found that in the European context, specifically in Italy “one in three children adopt passive responses to online risks that bother them, and one in four do not talk to anyone about what happened” (p. 145).

 Cyberbullying is one of the most important and studied online risks that youths may encounter. It is defined as an intentional aggressive online behavior led by an individual or group of people, using electronic devices against a victim who cannot defend himself/herself (Menesini et al., 2012, 2013; Smith et al., 2008). In the European context, youth involved in this phenomenon can reach values close to 10% (Smahel et al., 2020). The incidence of cyberbullying increases with age: in line with international research, the phenomenon is mostly present in adolescence than in the younger age groups (Livingstone et al., 2011; Smahel et al., 2020).

Understanding the relation between online risk awareness and behavior is important especially for preventing purposes. A study conducted by Letica (2019) found that high school students who have greater ICT safety awareness, take fewer risks. On the other hand, Sallán and Mercader (2018) have argued that risky behavior increases between 15 and 17 years. For this reason, they underline the importance of taking preventive action in earlier age groups.

Awareness, Digital Literacy and School Policy to Foster Prevention

By mentioning prevention, it is important to refer to models that can take into consideration different gradients of risk of involvement. In the multi-tiered model (National Academies of Sciences Committee, 2016), the first level is Universal prevention, which reduces risks and strengthens skills by targeting the full population within a defined community or setting. The second is Selective prevention, which may either target people who are at risk of engaging in risky behaviors or target youths at risk of being victimized. The last one is Indicated prevention that refers to actions devoted to taking care of those already involved in risky behaviors and who are showing the early signs of behavioral, academic, or mental health consequences.

To make the most of the opportunities offered by the ICTs and to avoid incurring possibly very serious risks, an essential aspect is to increase youth awareness on safe use, as one of the core universal prevention actions that can be carried out (Wurtele & Kenny 2016). Digital literacy and digital skills within Internet safety policy are crucial aspects (O’Neill et al., 2011). Scholars assert that digital skills can prevent online risks and help young people make the most of the opportunities offered by ICTs (Livingstone & Helsper, 2010; Mikhaleva, 2016; Rodríguez-de-Dios et al., 2018).

The European Commission in 2012 highlighted some key objectives in order to make the Internet a better place for youths. Raising awareness of positive use of ICTs seems crucial to European policy (Livingstone et al., 2011), and it is possible both through online and face to face initiatives (e.g., seminars, media campaigns, dedicated days) (O’Neill & Dinh, 2018). Among these, we can find raising awareness and youth empowerment; the online support for high-quality content for children and young people; creating a safe environment for children online; and combating sexual abuse and sexual exploitation of children. An interesting initiative in Europe is to try to integrate Internet safety into educational settings (O’Neill & Dinh, 2018). This goal can be reached through the development and promotion of training, awareness raising activities to improve digital literacy and skills, not only for young people but for all the agencies involved (e.g., school and family; Berger & Wolling, 2019; Savirimuthu, 2011; Stodt et al., 2016).

The adoption of an ecological approach is crucial. The scientific literature highlights how programs that target complex peer dynamics, such as bullying and cyberbullying, are more successful when they adopt a socio-ecological approach. In fact, their efficacy is related to the involvement of teachers, parents, and, in general, a school-wide method (Ttofi & Farrington, 2011). Similarly, prevention and intervention programs should target students along with the entire school community when they focus on promoting a positive online environment: teachers, principals, and parents, but even the broad community should be aware of both the risks and opportunities of ICTs. Modelling processes are crucial for the children’s development and online protection (Lo Cricchio et al., 2021). Some key aspects are having the possibility of establishing positive relationships between adults and children, so that they feel at ease asking for help if an online situation gets out of hand, and the promotion of a safe online space in which children and adults are part of the same community.

The definition of specific policies and guidelines for youth, teachers and parents is also highly recommended to prevent online risks and promote positive use of ICTs. This aspect is underlined by Forkosh Baruch and Erstad (2018), who affirm the importance of promoting specific projects and developing guidelines for digital skills to preserve the well-being of young people in a context of ICTs. The adoption of school policies about the use of ICTs is important in order to increase awareness of digital technologies, online risks (such as cyberbullying and sexting), opportunities of ICTs, and also to take care of any online incidents. Different studies found that ICTs innovation in education depends on the positive attitude and perception of teachers towards technologies (de Pablos Pons et al., 2010; Mura et al., 2014). In this regard, a study conducted by Tondeur and colleagues (2008) shows how adopting a school policy shared by teachers, promotes the integration of ICTs at educational level. For this reason, the implementation of school policies is important to strengthen the educational, positive uses of the ICTs (O’Neill, 2014).

However, it is not enough to just adopt projects that sound interesting or that are theoretically based to raise awareness of positive use of the Internet. It is essential to assess whether the actions chosen and proposed are effective in achieving the proposed objectives, by adopting a more evidence-based approach (Flay et al., 2005; Gottfredson et al., 2015). In fact, our study aims to evaluate the efficacy of the SIC project, which we will describe in the next paragraph.

The Safer Internet Centre – Italy

The SIC project is co-financed by the European Commission within the Connecting Europe Facility (CEF) Programme – Safer Internet. The focus of the present study is the 3^rd edition (years 2017–2018), a continuation of SIC I (years 2012–2014) and SIC II (years 2014–2016) project, improving and strengthening the role of the SIC project, as a national reference for issues related to online safety, youths, and positive media use.

The actions carried out by SIC project are based on three levels of prevention (Preventing Bullying through Science, Policy, and Practice, 2016):

• Universal, through awareness raising initiatives (e.g., media campaigns to raise knowledge and develop strategies to stay safe online and take advantage of opportunities; online materials and activities to support schools in developing and implementing their own policy of online safety – ePolicy);
• Selective, intervening with face-to-face activities in schools at higher risk of unsafe use of ICTs;
• Indicated, thanks to a Helpline that gives advice and support to children and adults on issues related to children’s safety in the digital world, and Hotlines to report online child pornography material.

The core action among the Universal prevention strategies is to empower middle schools on the development of their own ePolicy. This document, shared and approved by the school council, underlines what the school does (and wants to do) in order to prevent and counteract online risks and promote a positive use of ICTs among students, teachers and parents. The first step of the intervention is a self-evaluation of the strengths and weaknesses made by each school taking part in the project. A Self-Evaluation Questionnaire is compiled by teachers and principals at the beginning of the school year. This evaluation determines whether the school needs an additional dosage of intervention.

The questionnaire was an ad hoc instrument developed in a previous edition of the project and takes into account the specificities of the country in which the project was carried out. It is comprised of 23 items, and it delves into the school situation in 4 areas:

• Part A: Digital skills and ICT integration in education – this includes teachers' knowledge of digital technologies and the use of technologies in educational activities (i.e., How many teachers are familiar with ICT in general terms and use it to supplement teaching?);
• Part B: Value system and attitudes – investigates how the school promotes a climate of respect for diversity involving students and parents (i.e., Our school is engaged in projects and events aimed at students to promote education in affectivity and sexuality);
• Part C: Risks of unaware use of ICT and prevention – concerns the knowledge of risks, prevention methods put in place to counter online issues (i.e., Each school must have a curriculum for the development of digital skills – understood as a set of knowledge, skills and attitudes);
• Part D: Incidents related to online risks: it focuses on how the school manages incidents and monitors online issues over time (i.e., Each school must detect and monitor, over time, the problems arising from an unconscious and responsible use of the Internet and digital technologies – e.g., Cyberbullying, Sexting, Privacy Violation, Online Grooming, etc.);

Schools with a low score on the Self-Evaluation Questionnaire (i.e., a score lower than 40, the 25^th percentile of the distribution) are defined as High Risk schools and receive a higher dosage of intervention. Schools with a higher score on the Self-Evaluation Questionnaire receive a lower dosage of intervention (i.e., just online) and, for the purposes of the present study, if they had a score in the 75^th percentile of the distribution (i.e., over 160), were defined as Low Risks schools.

The aim of the schools is to develop and implement their own policy of online safety. The schools considered less problematic receive a lower dosage of intervention consisting of online resources, materials, activities and support (e.g., a devoted email to support schools in the path of ePolicy development; e-learning courses for teachers to support them in developing the school ePolicy; a cartoon video – SuperErrori campaign and other materials to carry out awareness raising activity about ICTs risks and opportunities; specific areas dedicated to parents, children, and teachers on the project website; training webinars dealing with various issues of online security).

The schools considered more problematic, receive an additional face-to-face component (i.e., higher dosage—selective prevention actions): a 2-hour meeting with students and a 2-hour meeting with their parents to raise awareness on online risks and opportunities; a 3-hour meeting with teachers, to support them in developing the school ePolicy.

Purpose of the Present Study

Our study aims to evaluate the effects of the SIC Italy project school activities on (a) students’ awareness of online risks; (b) students’ perception of the schools’ policy both on ICTs risk prevention and incidents; and (c) cyberbullying (i.e., both victimization and perpetration). Adopting an experimental matching design in two consecutive school years, two trials of intervention were carried out with different samples.

In the first trial of intervention, experimental and control middle schools were compared on the three outcomes. In the second one, we compared two different experimental groups: middle schools that received a higher dosage of intervention (i.e., online, and face-to-face components) because of their lower level at the preliminary self-evaluation; middle schools receiving a lower dosage of intervention (i.e., only online) because of their better starting situation.

Our hypotheses concern the following aspects related to the core aims of the project:

H1: Effects directly on the students: the program is effective in raising awareness about ICTs risks .

H2: Effects on the school context perceived by the students: the program is effective in promoting the adoption of a school policy to prevent ICT risks and incidents.

Regarding cyberbullying, we know from previous literature about the reduced efficacy of programs targeting this behavior, how difficult it is to change peer dynamics, even online: the results of a recent meta-analysis indicates that anti-cyberbullying programs can reduce cyberbullying perpetration by approximately 10%–15% and cyberbullying victimization by approximately 14%. Consequently, hypotheses about the effects of this behavior, that are not directly and intensively targeted by the project, remain open.

Methods

Participants and Data Collected

The schools considered in the following studies are first grade secondary schools (i.e., middle school). The participating students attend from 6^th to 8^th grade. Specifically, the Italian school system is organized into two levels of education: primary school (i.e., elementary school), which lasts 5 years (1^st to 5^th grade) for children between 6 and 10 years old; secondary school, composed by the first grade (i.e., middle school), which lasts 3 years (6^th to 8^th grade), for children between 11 and 13 years old, and the second grade (i.e., high school), which lasts 5 years (from 9^th to 12^th grade) for youths between 13 and 19 years old.

In the first study, the sample was composed by 7 experimental schools (randomly selected within all Italian schools participating in the project)—Higher dosage, paired with 7 control schools on the basis both of the province and their general characteristics (i.e., size in terms of students and area—city or countryside).

Data were collected during the school year 2016/2017 at the beginning of the school year (pre-test) and at the end (post-test). The experimental group was composed by 775 students, 47.2% female with an average age of 12.13 years (SD = 0.94). The control group was composed by 675 students, (45.9% female), with an average age of 12.17 years (SD = 0.91).

In the second study, the sample was composed of two experimental groups: 4 higher dosage schools were randomly selected and matched with 4 lower dosage schools. Lower dosage schools had a score on the Self-Assessment Questionnaire ranging from 160 to 238, while higher dosage schools had a score on the Self-Assessment Questionnaire ranging from 26 to 38.

Data were collected during the school year 2017/2018 at the beginning of the school year (pre-test) and at the end (post-test). The higher dosage group was composed by 251 students, 46.2% female with an average age of 12.06 years (SD = 0.98); while the lower dosage group was composed by 315 students, 45.1% female, with an average age of 12.3 years (SD = 0.87).

Measures

The same questionnaires were administered to students in every group both at the pre-test and at the post-test.

ICTs Risks Awareness

The ad hoc scale evaluates awareness of online risks. It consists of 4 items (i.e., I know what cyberbullying is; I know what online grooming is; I know what sexting is; I know what online privacy is). Each item was evaluated along a scale from 1 to 10. The scale shows acceptable levels of internal consistency in both trials pre/post evaluation (Cronbach’s Alpha was, respectively, .60 (pre-test) and .67 (post-test) in the first study and .60 (pre-test) and .64 (post-test) in the second study.

School Policy on ICTs Risks Prevention

School policy on ICTs risks prevention evaluates the school policy on risk prevention from the students’ perspective. It consists of 3 items (In your school, they explained how to use digital technologies in a safe way; In your school, there are clear rules on how to use the Internet during schooltime; In your school, it explains how to avoid dangerous situations while using digital technologies). Each item was evaluated along a 4-point scale (i.e., false; not very true; true; very true). The scale shows acceptable levels of internal consistency in both trials pre/post evaluation (Cronbach’s alpha was respectively .69 (pre-test) and .71 (post-test) in the first study, .70 (pre-test) and .73 (post-test) in the second study.

School Policy on ICTs Incidents

School policy on ICTs incidents evaluates the school policy on ICTs incidents. It consists of 3 items (In your school, adults notice if something bad happens to the students online; The school has clear rules on how to deal with ICTs incidents (i.e., cyberbullying); If any boy or girl has a problem online, your school has an adult to help him or her). Each item was evaluated along a 4-point scale (i.e., false; not very true; true; very true). The scale shows acceptable levels of internal consistency in both trials pre/post evaluation (Cronbach’s alpha was, respectively, .63 (pre-test) and .67 (post-test) in the first study, and .67 (pre-test) and .70 (post-test) in the second study.

Cyberbullying Involvement

We used two items to evaluate involvement in cyberbullying (Have you taken part in cyberbullying episodes against other students within the last 2–3 months?; Have you been cybervictimized by your peers within the last 2–3 months?). The items were presented after a definition of what cyberbullying is (Menesini et al., 2012) that is currently used in the introduction of an already validated questionnaire to assess cyberbullying (Palladino et al., 2015). The definition stressed the use of ICTs, provided examples of cyberbullying behaviors (Palladino et al., 2017) and criteria such as imbalance of power, intentionality, and repetition, adapted to the online context (Menesini et al., 2013). This is a commonly used approach to measure cyberbullying that follows the Olweus Bullying questionnaire (Olweus, 2007). Although this approach could have critical aspects (Menesini & Nocentini, 2009), it has the clear advantage of a reduced questionnaire which takes less time to fill in. Each item was evaluated along a 4-point scale from never to very often.

Procedure

 Each school was contacted first by email. The informed consent form (according to current legislation) was sent to be signed by both parents (a necessary requirement to participate in the study). The questionnaire was administered by trained teachers in two ways: paper and pen, or online. The method of administration was defined following the needs and characteristics of the schools (e.g., in some schools there were no available laptops or tablets connected to the Internet). For schools in the experimental group, the questionnaire was administered prior to the beginning of the intervention. During the same period, administration was also carried out in control schools. In both studies, the same questionnaire was administered again five months later, at the post-test data collection.

Data Analysis

Altogether, four separate models were estimated for study 1 (outcome variables: ICTs risks awareness, school policy on ICTs risks prevention, school policy on ICTs incidents and cyberbullying involvement) and five for study 2 (outcome variables: ICTs risks awareness, school policy on ICTs risks prevention, school policy on ICTs incidents, cyberbullying and cybervictimization).

In the first study we used Structural Equation Modelling (SEM) to analyze the effects of the group (control = 0 vs. experimental-higher dosage = 1), separately, on three out of four outcomes measured at the post-test controlling for the pre-test evaluation: ICTs risks awareness, school policy on ICTs risks prevention and ICTs incidents. Given that cyberbullying involvement was measured by two single items (perpetration and victimization), in this case we performed a path analysis with both variables tested in the same model. SEM were used to include the measurement part into the models. This allows us to take into account the measurement error especially because we used ad hoc developed questionnaires.

In every tested model we considered the nested nature of the data by using the “stratification” procedure for the pairment we used between the experimental and control schools. The analyses were conducted with Mplus 7.0 (Muthén & Muthén, 2012). We used the maximum likelihood estimator. This method uses all the data that are available to estimate the model without imputing the data. The following fit indices were considered overall for the SEM models evaluation (Hu & Bentler, 1998): chi-squared although it is well recognized that it is highly sensitive to the sample size (Schermelleh-Engel et al., 2003); CFI > .95 optimal, > .90 acceptable; RMSEA—optimal fit when less than .06; and SRMSR—values below .06 indicate a good model fit.

In the second study we used repeated measures ANOVAs to test for differences between the two groups (between-subject factor) over time (within-subject variables) in the four outcomes. Effect size was evaluated by mean of partial eta squared (above .02 = small, above .13 = medium, above .26 = large; Cohen, 1988). The analyses were conducted with SPSS 25.0 (IBM Corp, 2017).

Results

Study 1

ICTs Risks Awareness

The model shows good fit indexes, χ²(22) = 148.148, p < .001, RMSEA = .063, 95% CI [.054, .073], CFI = .932, SRMR = .045. We found a significant effect of group on ICTs Risks Awareness measured at post-test, β = .250, SE = 0.033; p < .001; the experimental—higher dosage group showed a significantly higher level of ICTs Risks Awareness compared to the control group. We also found a significant negative small effect of group on the pre-test, β = −.096, SE = 0.035, p = .006). The standardized estimates of the model are reported in Figure 1.

School Policy on ICTs Risks Prevention

The model shows good fit indexes, χ²(12) = 68.536, p < .001, RMSEA = .057 [.044, .071], CFI = .954, SRMR = .033. We found a significant effect of group on School policy on ICTs risks prevention measured at post-test, β = .119, SE = 0.035, p = .001; the experimental—higher dosage group showed a significantly higher level on this variable compared to the control group. We also found a significant negative effect of the group on the pre-test, β = −.133, SE = 0.033, p < .001, emphasizing how the experimental group showed a significantly lower level of this variable before the beginning of the intervention. The standardized estimates of the model are reported in Figure 2.

School Policy on ICTs Incidents

The model shows good fit indexes χ²(12) = 90.619, p < .001, RMSEA = .067, 95% CI [.055, .081], CFI = .917, SRMR = .042. We found a significant effect of group on School policy on ICTs incidents measured at post-test, β = .091, SE = 0.042, p = .029; the experimental—higher dosage group showed a significantly higher level on this variable compared to the control group. We also found a significant negative effect of the group at pre-test, β = −.243, SE = 0.037, p < .001, underlining that the experimental group initially showed a significantly lower level of this variable. The standardized estimates of the model are reported in Figure 3.

Figure 1. Standardized Estimates and Standard Errors (in Brackets) of the Model Tested for ICTs Risks Awareness.

Note. Significant paths between the group variable and the latent variables are in bold. Residual variance of the outcome variable is in bold. Standard errors are reported in brackets. Group: control (0) vs. experimental (1).

Figure 2. Standardized Estimates of the Model Tested for School Policy on ICTs Risks Prevention.

Note. Significant paths between the group variable and the latent variables are in bold. Residual variance of the outcome variable is in bold. Standard errors are reported in brackets. Group: control (0) vs. experimental (1).

Figure 3. Standardized Estimates of the Model Tested for School Policy on ICTs Incidents.

Note. Significant paths between the group variable and the latent variables are in bold. Residual variance of the outcome variable is in bold. Standard errors are reported in brackets. Group: control (0) vs. experimental (1).

Cyberbullying and Cybervictimization

The model shows good fit indexes, χ²(2) = 16.962, p < .001, RMSEA = .072 [.043, .105], CFI = .983, SRMR = .043. No significant effects of the group have been found on cyberbullying nor on victimization at any time. The standardized estimates of the model are reported in Figure 4.

Figure 4. Standardized Estimates of the Model Tested for Cyberbullying and Cybervictimization.

Note. Residual variance of the outcome variable is in bold. Standard errors are reported in brackets. Group: control (0) vs. experimental (1).

Study 2

In Table 1 descriptive statistics about the outcome variables are reported for both experimental groups, higher and lower dosage both pre- and post- test. Repeated measures ANOVAs results showed only a significant main effect of time on the awareness on ICTs risks, F(1, 551) = 149.007, p < .001, η²_p = .21. Similarly, we found a significant main effect of time of school policy in terms both of preventing ICTs risks, F(1, 554) = 69.541, p < .001, η² = .11, and awareness of ICTs incidents, F(1, 550) = 15.288, p < .001, η² = .03. No significant interactions were found between time and group on school policy in term both of prevention, F(1, 554) = 2.498, p = .122, and ICTs incidents, F(1, 550) = 1.375, p = .241. No significant main effects of time on cyberbullying, F(1, 553) = .157, p = .691, nor cybervictimization, F(1, 553) = .405, p = .532, have been found. No significant interactions were found between time and group on cyberbullying, F(1, 553) < 0.001, p = .991, and cybervictimization involvement, F(1, 553) = .003, p = .962.

Table 1. Descriptive Statistics Pre-Post Intervention in Both Experimental Conditions
(Lower and Higher Dosage).

Discussion

The current study evaluates the efficacy of the SIC Italy project. The general aim of the project is to raise awareness of online risks and opportunities. In the present study we focused on one of the prevention actions (universal and selective) carried out at the school level: supporting the schools in developing and implementing their online safety policy (ePolicy) on ICTs risks prevention and intervention by Italian schools. In Europe, similar projects have been built up in each country thanks to European Commission grants since 2012 but, to our knowledge, no scientific evaluation of their efficacy has been published yet.

Using data of two consecutive trials of implementation, we found that the project was effective in raising awareness of the online risks: the experimental group shows a significant increase in the student’s perception of knowledge about cyberbullying, online grooming, sexting, and online privacy, with a medium effects size in the second trial. Teachers implemented a set of activities to raise awareness in students on online risks, listed in their own ePolicy. Conversely, no significant change over time has been found in the control group. This is in accord with the literature that emphasizes the importance of promoting activities within the school context in order to raise awareness of the risks and opportunities of digital technologies (Berger & Wolling, 2019; Forkosh Baruch & Erstad, 2018; O’Neill & Dinh, 2018).

The SIC Italy project aims to support schools in developing their ePolicy of online safety that should mention how the school aims to prevent and deal with online issues related to risks that can occur to students, teachers etc. The results show that, in both studies, from the students’ perspective, the project was effective in changing the school policy both on ICTs risks prevention and ICTs related incidents handling. As the literature shows (Milosevic & Vladisavljevic, 2020; Muls et al., 2020), adopting a school policy is important to allow the entire school community to have specific guidelines to improve their approach to online safety and positive use of digital technologies.

Our results are especially relevant in the current historical period, where the use of ICTs increased as a consequence of the measures taken by governments to counteract the spread of the COVID-19 pandemic. Digital technologies became a fundamental part of school education that allowed students to continue their studies during the lockdowns. The integration of ICTs in youths’ daily life has become massive, and consequently, it is crucial to sensitize young people, schools and family to a positive use while also trying to raise awareness on possible online risks.

Looking at the students’ online behavior, the analysis did not show any effect of the program on cyberbullying and cybervictimization. We can argue that improving awareness does not lead straight to a shift in behavior. We know that cyberbullying is a complex dynamic that results from the interplay between individual characteristics and contexts, and it is not easy to change it simply on the basis of an increased awareness. From studies directly addressing cyberbullying, the rate of success was approximately of 10–15% for both cyberbullying and cybervictimization (see the metanalysis of Gaffney et al., 2019), thus it is coherent that also our project has showed difficulties in changing the dynamics. It should also be noted that reducing cyberbullying was not the main objective of the project, which, instead, aims to raise awareness and attract attention to online risks, along with promoting positive use of ICTs. Probably, new specific components need to be added to the program in order to have effects on the behavior, by acting on specific mechanisms (e.g., group dynamics, peer support, face to face and online).

Looking at the selective preventive actions (i.e., online, and face-to-face activities for higher risk schools—Higher dosage group), no differences have been found over time in the comparison to the Lower (standard) dosage of intervention (i.e., online activities for lower risk schools). We can hypothesize that it was due both to the impact and the amount of the online resources, materials, and support provided by the project, compared to the higher dosage component. The latter was based on just three meetings (a full day of activities): an awareness meeting for students and another one for parents (two hours each) plus a three-hour long meeting with teachers, on the contrary, both groups Lower and Higher dosage shared all the online materials.

Another important aspect that needs attention is how higher and lower risk schools were classified. In the second study, no differences were found between schools with Lower and Higher dosage of intervention at pre-test from the students’ perspective. We can hypothesize that a mixed approach that takes the point of view of all the school components into consideration (i.e., students, teachers, principal, parents) could be more useful in order to understand the real situation of the school. The method used in the present study (i.e., evaluation of teachers and the principal) was unable to differentiate the schools—or, at least, was unable to catch differences consistent with students’ perspective.

Limitations

Some limits of the present study should be noted. The first concerns the absence of a follow-up measure to evaluate the stability of the effects found and to test if some effects need more time to show up (e.g., on cyberbullying, a complex peers’ dynamic).

Moreover, a direct comparison between Control and Higher and Lower dosage groups would be more informative and specific, avoiding the risk of confounding effects in our grouping. Unfortunately, in the first study, a limited number of schools in the Lower dosage condition were present, and consequently, it was not possible to randomly pair the Higher dosage schools with them on the bases of their geographical collocation (i.e., province). For this reason, a new independent trial was carried out focusing on the dosage effects but also aimed at confirming the efficacy of the project on its core variables.

A third aspect of limitations concerns the measures used. The use of single items to assess cyberbullying was appealing for us in order to reduce the number of items administered. However, future studies should consider using a validated scale that evaluates the frequency of different behaviors—items, avoiding issues related to social desirability of respondents and having a more valid measure of the construct. Additionally, being aware of “what an online risk is” is only the first step of a process which leads to full awareness of the risk itself and concludes with the promotion of positive prevention strategies to keep the individual safe (i.e., behavior). In our study we measured students’ perception of knowledge of risks. Our results underlie how complex the transition from gaining a new understanding, to changing one’s behavior (see the stability of cyberbullying) is, and how this aspect should be kept in consideration when developing and implementing intervention programs. Finally, we want to suggest that future studies should use the continuous measurement of the Self-Evaluation Questionnaire as a control variable in analyses. Specifically, using a different approach that includes a continuous measure of the risk level as a covariate (i.e., the Self Evaluation Questionnaire score) in the research design, instead of grouping schools, will reduce the impact of possible confounding effects.

Finally, we acknowledge that the teachers’ point of view is fundamental within the school context, especially when we are talking about interventions to promote the safe use of digital technologies (Mura et al., 2014). On the other hand, it is important in future studies about project evaluation to include the parents’ perspective, especially because cyberbullying can happen outside of school, and it would be important to analyze the parents’ perspective of what students do.

Conclusion

Conducting two different studies allowed us to take into account contextual variables (e.g., related to the geographical area in which the school is located) to assess the efficacy of the intervention, as a first step, and then delve deeper into aspects related to the dosage of the intervention. In the first study the two groups were defined on the bases of absence (control group) or presence (experimental group) of intervention. In the second study both groups were experimental: a higher dosage group was randomly selected and paired with the lower dosage group. The second study allowed us to evaluate the effect of the dosage of the intervention and to confirm the effectiveness of the project on the core variables evaluated in the first study. Although more research is needed to disentangle whether the face-to-face component was able to booster the efficacy of the project, or whether the tool to assess the school risk level played a confounding role, the second study offers insight on important cost-effect implications for the project, an aspect which needs more attention, even with a more rigorous approach.

In conclusion, the present study represents the first evaluation of the effectiveness of the awareness raising actions promoted at school by the SIC project in Italy. The evaluation is important not only to define if the intervention matched its aims but also to decide further improvements in the future. Specifically, in our case, looking at the results, we can state that the online components of the project were particularly effective. Consequently, it is highly encouraged that the project keeps investing in them, in order to be effective in raising students’ awareness on ICTs risks and promote a school environment that takes care of the prevention and management of ICTs related incidents. Future directions for improving the SIC Italy project could focus on changing schools’ initial screening to propose more effective selective prevention actions and add specific components targeting cyberbullying.

Conflict of Interest

The authors have no conflicts of interest to declare.