Is it possible to develop social-emotional skills of adolescents? Evidence from Brazilian Program

There are scarce but relevant researches stating the importance of educating socioemotional competencies to the youth as it impacts individual’s success in life, and what is even more scarce is the analysis of the results and impact of programs that work towards that educational goal, either implemented though public policies and Government funded projects. This paper seeks to evaluate the impact of the Academia Educar Project in 2016, focusing on its’ development of socioemotional skills on students using the methods of Propensity Score Matching and Differences in Differences. Positive and significant effects were found on Sociability (12% of initial value), Assertiveness (16% of initial value) and Political Participation (double the initial percentage of treaties interested in participating in the country's policy). The results for Locus Control and Imaginative variables were significant and in the direction expected only in part of the specifications; for Volatility, the results do not suggest impact. Several robustness analyzes were performed to validate the results found.


Introduction
The relevance of the present research can be understood based on two distinct factors: as it analyses the impact of a program it opens precedent for other programs to be analyzed and the reflection upon certain results indicate best ways to move forward, knowing what might be more effective in terms of its educational goals. Adding to that, we must point out how relevant it might be to a society the subject of this: the education and future success in life of the youth, which might be enhanced through the development of socioemotional skills.
Government policy implementations carried out by State Governments and Non-Governmental Organizations need to have a clear evaluation of results to use their resources more efficiently. Despite its importance, the evaluation of public policies is still not that common, although it has been growing recently. (MENEZES-FILHO, 2012). This research is highly focused on understading results through data, and applying this analysis in the future to further improve the program.
Academia Educar is the first project developed by Fundação Educar DPaschoal and began to be idealized in 1989. 1 The project promotes the training of young leaders in public schools, creating opportunities for 13 to 16-year-olds to discover their natural capabilities, empowering them to transform the reality of their schools and communities. The project is based on the four pillars of education established by Unesco: Learning to Be, Learning to Live, Learning to Learn and Learning to Do.
For the evaluation, two field surveys were carried out. First, the diagnostic questionnaire was applied to all students in the treatment and control group. This initial research, conducted in March 2016, aimed to know both the socioeconomic characteristics of each student and their cognitive and socioemotional skills. The second round of questionnaire was taken in November of the same year to collect only information related to the students' socioemotional and cognitive competences. On both occasions, the students were responsible for completing the questionnaires applied.
An experimental impact assessment is likely to produce more reliable estimates, although non-experimental methods of public policy evaluation are widely used to understand the causal relationship between socio-emotional skills and various outcomes. The international literature has relatively scarce studies and interventions through experimental 2 designs whose purpose is to investigate the impact of educational programs. According to Primi and Santos (2014), there are also few analyzes of the impact of policies and programs that seek to promote individual and collective success through the development of socio-emotional skills. Due to the non-random character of the Educar Academy project, quasi-experimental 3 techniques are used to identify the impact of the program on social-emotional outcomes.
The present study seeks to evaluate the impact of the Academia Educar program in 2016 in public schools in a region of the state of São Paulo. To estimate the causal effect, two control groups are considered, in order to mitigate possible estimation biases. The first control group consists of students from a school that does not have students belonging to the program, 186 individuals from the State School (EE) Profª. Maria Julieta de Godoi Cartezani. This control group has the advantage of avoiding the contagion caused by the students of the treatment group, that we will discuss in section 2. The second control group is formed by students from one of the participating schools, 41 individuals from EMEF 4 Odila Maia Rocha Brito; totaling 227 students. This second control group is used as a way of testing the robustness of the results.
To investigate the effect of the Academia Educar program on impact indicators, the students selected to receive the intervention in 2016 (treated) with the students who did not receive the intervention (control) were compared. For this, a combination of the propensity score method with Differences in Differences 5 is used. Several robustness analyzes were performed to validate the results found. In general, the results indicate positive and significant impacts on the Sociability, Assertiveness of students and on the interest in participating in national politics. For the Locus Control and Imaginative variables, the results were not conclusive. For the variable Volatility, in all estimates, the results did not indicate that the program had an impact.
All this program analysis focuses primarily on the understanding of improvements on perceived socioemotional skills as it lays upon the premise that those skills can be a proxy to individuals success, which relates to the Academia Educar's own objective and is a matter of investigation in researches as we can better explore in the upcoming paragraphs. 2 For experimental studies, see Ramey (1974), Schweinhart (2004), Bloom, Gardenhire-Crooks and Mandsager (2009), Gottschalk (2005), Dee and West (2011) and Chetty et al. (2011). 3 For studies with quasi-experimental techniques, see Hong and Yu (2008), Fletcher and Wolfe (2016), Junior and Gonçalves (2016). 4 Municipal School of Elementary Education. 5 For further details, see Heckman, Ichumura andTodd (1997, 1998).
There is a meager amount of impact analysis of programs that seek to promote individual and collective success by developing socioemotional skills. On the other hand, several studies examine the effects of cognitive ability, such as Farkas et al (1997), Murnane et al. (2000), Kerckhoff, Raudenbush andGlennie (2001), Farkas (2003), Riani and Rios-Neto (2008). Recently, researchers have begun to investigate the role of non-cognitive skills and their applications, Sarzosa and Urzúa (2015) and Oliveira et al. (2018) investigate how social-emotional skills help mitigate the effect of bullying. Cunha et al. (2010) show that individuals with more non-cognitive skills in childhood are more likely to increase their cognitive indicators and Gensowski (2012) shows how lifetime financial gains are influenced by education and personality traits. Lleras (2008) shows that non-cognitive behaviors measured in high school have significant effects on later educational achievement and higher income. These effects appear to be as important and perhaps more important than cognitive abilities in determining outcomes. However, the benefits of skills and behaviors are different from the groups. For example, non-cognitive behaviors seem to play a more important role in explaining the female and Asian advantage in educational attainment compared to cognitive ability.
Many researchers have attempted to understand how academic performance and cognitive ability have determined social performance and schooling. Some early researchers point out the importance of non-cognitive skills, as good predictors of success in life were Bowles and Gintis (1976).
According to Carneiro, Crawford and Goodman (2007), economists often have a simplified view on how socioemotional skills act as the determinants of economic and social outcomes. This is because these abilities are intrinsically multidimensional. This study also identifies how non-cognitive skills affect the possibility of young people smoking at age 16, health status at age 42, employment status at age 42. The work suggests that non-cognitive skills appear to be more malleable than cognitive abilities. If this is true, an education policy that focuses on noncognitive skills may be more effective at generating increased social and economic outcomes than one that targets only cognitive abilities. Almlund et al. (2011) also consider more malleable personality traits throughout the life cycle than cognitive ability, which becomes highly stable at around 10 years of age. His work suggests that interventions are capable of changing personality traits and promising ways to address poverty and disadvantage.
A survey conducted by the National Center for Workplace Education (NCEQW) in 1995 asked employers to rate some characteristics or attributes regarding their criticality in the hiring decision. Employers tend to minimize schoolbased factors in making their decisions. Characteristics and attributes, such as the candidate's attitude, communication skills, previous work experience, and cur-rent employers' recommendations, seemed to be more important than full years of schooling, test scores, academic performance and teacher recommendations (STASZ, 2001). In 1997, the NCEQW achieved the same result, suggesting that the view on skills persists over time, even when economic conditions and overall labor demands fluctuate, as Shapiro and Goertz (1998) argue. Heckman, Stixrud and Urzua (2006) use data from the National Longitudinal Survey of Youth to verify that non-cognitive skills are at least as important as cognitive abilities to explain some social performances in life. For example, noncognitive skills seem to have a strong influence in making decisions about school choice, work and occupation. The authors point out the importance of including these skills in explaining the likelihood of one engaging in risky behavior.
There are different options when trying to measure the personality traits of an individual. Mischel, Shoda and Rodriguez (1989) used an experiment called the Marshmallow Test, which showed that 4-year-old children with greater ability to postpone rewards tend to be smarter, more likely to have greater social responsibility, and postponement time is significantly related to higher grades in the SAT. 6 These results suggest that the relationship between the delaying the reward and the ability to cope with a range of social and personal problems is not entirely attributable to schooling. Primi and Santos (2014) investigated the description of the socioemotional profile of students in Rio de Janeiro -Brazil, and Soto et al. (2011) investigated the profile of students in various places around the world and found similar results. In the above-mentioned studies, the data reflect that girls tend to be more conscious, extravagant and enjoyable. On the other hand, girls had less emotional stability. Both group of researchers used the "Big Five 7 Model", which was suggested as relevant for measuring personality traits in the context of education by Kyllonen et al. (2008). This same model is in parts used in this research to understand students' characteristics and development throughout the process.
In addition to this introduction, a brief description of the Academia Educar project is presented below. Section 3 provides descriptive analysis of data from Academia Educar. Section 4 presents the identification strategy used and section 5 the results obtained. Finally, the last section brings the final considerations of the paper.

Description of the Academia Educar
The project works in partnership with 20 schools (10 municipal and 10 state) and has support from the Departments of Education East and West and Municipal 6 SAT (Scholastic Assessment Test) is a standardized test widely used for college admissions in the United States. 7 It refers to the five personality factors. For more details, see McCrae and John (1992).
Education Secretariat of Campinas-SP, receiving about 100 direct participants, 5 students from each of the schools. In the Academia Educar, teenagers exercise their mind in workshops that happen twice a week, out of school hours (one class in the morning and another in the afternoon). They also develop, once a week, the multiplications, that is, workshops from 1h to 2h for other students of the schools participating in the project. These workshops have a responsible educator who helps in the performance of the students. In addition, there are some challenges developed throughout the year that contribute to the development of technical knowledge and social-emotional skills. The Academia Educar project wants to foster Youth Protagonism, wants to make them know their rights and duties as a citizen. The goal is for young people to be able to transform their reality, starting with their school and evolving all the way to reach their community. The target audience of the program is precisely the students of the public schools of Campinas, aged between 13 and 16 years who are studying from the 9th year of Elementary School II to the 2nd year of High School.
Several workshops are held with these young people with the aim of developing their leading role in their own context. The idea is that young people are able to lead projects that foster their community's development. In addition to the training, the participants are challenged and commit to carry out 8 projects engaging the entire community and replicating the knowledge acquired to the other students of the school, throughout the year. With respect to these projects at school, the idea is for the youngsters to identify needs within their schools and then come up with projects to address these needs. For example, if there is a problem of difficulty in reading in the school, the youngster may suggest Reading, Storytelling workshops and so on.
The project lasts one year (school year). There are 250 hours and 102 hours of training (which take place at DPK 8 Campinas, twice a week -Tuesdays and Thursdays, within 3 hours) and 148 hours, 148 hours dedicated to the production of the 8 challenging projects: Multiplication in school; Station Experience; Oasis Educar; Catavento de Letras; Catavento de Números; Knowing the world; School Project and Closing Show.
The State Department of Education delegates to the Directorates West and East the indications of the schools seeking to contemplate the various districts of each region and also receive indication of desire for renewal manifested by the Educar DPaschoal Foundation of the schools that have participated in previous years. The Municipal Secretariat sends a communication via e-mail to the municipal schools and those that express interest are contemplated by order of arrival.
Once the participating schools have been defined, they must indicate who will be the teacher of the school in the Educar project. There is always a school educator who acts voluntarily. In the Academia Educar, the educators of the schools are a core part in the process of development of the juvenile leadership. Each school has an educator that supports the project. Participating schools must share with the values of the Academia Educar.
At Academia Educar, the idea is to help develop youth to become leaders who are capable of initiating change and one of the keys to success in the project is in the presence of young tutors. 9 These young people, after passing through the development period as students, are invited to become tutors for the next batch of students in the following year. They hold weekly meetings to create the workshops according to the proposed theme. In addition, each tutor "sponsors" some schools to follow the multiplications with the students, giving tips and enhancing the development of the beginners and the results obtained in the project.
Another important point in this process is the choice of students who will participate in the program. First, staff at the Educar DPaschoal Academy team announces the program to their students. Afterwards, the young people interested participate in a group dynamic, conducted by the technical staff of Academia Educar, which lasts from three to four hours and takes place in school. From this group dynamic, the Educar DPaschoal Foundation team and the young tutors select which students will join the Academia Educar of the year.
The eligibility criteria are: to be enrolled between the 9th year of the high school and the 2nd year of the middle school in public schools that adhered to the project in the current year and between 13 and 16 years old. However, during the group dynamic, the technical staff of the Academia Educar mainly observes the interest and willingness of the student to participate. Secondly, they also observe the posture under pressure, initiative, cooperation / teamwork, commitment, flexibility, among others. It is based on these abilities that the team decides who will be selected. In short, criteria such as the commitment and willingness of young people to participate are major deciding factors.

Descriptive Analysis
In this section, we describe the data used for the impact assessment of the Academia Educar 2016 project. We present the information regarding the treatment group and then those of the control group. In addition, a friction analysis is performed. Finally, a subsection presents a simple comparison of the treatment and control groups in the pre-intervention and post-intervention periods.
An important analysis is the friction of students in the treatment and control group. Friction is the loss of observations (relative to the initial design) that occurs in a field survey. Every program has the objective of treating all its candidates, however, a part of the candidates leaves the treatment for several reasons. Thus, when we return to the field to interview students after treatment, a portion of the initial sample is lost. The same may occur for the control group. In the case, we had a friction of 25% in the treatment group and 21.14% in the control group -these numbers are precisely the difference between the ideal response rate (without friction) and the observed response rate.
Inferences may be biased if this friction is not random, that is, when individuals who did not respond to the (second) survey are different from the group as a whole. To verify that the field friction did not mischaracterize the initial sample, generating a sample selectivity, are compared the initial treatment and control groups with the sample that did not respond to the final field survey using the variables collected at the initial time of the Academia Educar survey.
The friction analysis showed that the treaties that left the Program do not differ from the total initial sample. In terms of their observable characteristics they showed no significant divergence at 90% confidence. This suggests that the friction may have been random for the treated students. For the students in the control group, the only exception was the variable Volatility was statistically significant (10% significance when the control group was formed by the students of the school in which there was no treaty and 5% of significance when we grouped the youngsters of the two schools to form the total control group).
In the pre-treatment period, the participants of the Academia Educar project are the students of the 9th year of Elementary School II, 1st and 2nd year of High School, respectively 49.3%, 46.6% and 4%. 10 The students in the treatment group are 36% boys and 64% girls. According to Table 1, the average age of the boys is around 14 years, with a minimum of 13 and a maximum of 16 years.
Regarding schooling of the student's parents, 61.3% of these parents present a level of schooling up to the 5th year of Elementary School, 28% of them have completed High School and 10.6% did not specify. Regarding the level of schooling of the mother, 62.6% have a level of education up to the 5th year of Elementary School, 34.6% have completed High School, 2.6% did not specify. In relation to family income, 93.3% have an income between R $ 0.00 and R $ 3,000.00.
Regarding the students in the control group, Table 1 shows that 6.7% of the young people are 8th grade students, 44.6% are 9th year of Elementary School II and 48.6% of students are attending the 1st year of high school. The control group consists of 47.5% girls and 52.5% boys. The average age of young people is around 15 years.
The schooling of the parents is composed of 88.3% up to the 5th year of Elementary School and 10.6% present the complete High School. The level of education of the mother up to the 5th year of elementary school is 85.4% and with high school is 14.5%. The family income is formed by 79.8% with income between R$ 0.00 to R$ 3,000.00, and with income over R $ 3,000.00 is 20.1%.
As previously explained, the data collection for the evaluation occurred in two phases. At the first moment (baseline), the students answered a questionnaire about their socioeconomic characteristics and socioemotional competences. After the intervention of the program, the students of the treatment and control group answered again the questionnaire about their social-emotional abilities. For constructions of the facets 11 were considered only the students who answered the questionnaires before and after intervention of the Educar Academy. Moreover, since the facets are based on several items in the questionnaire, an incomplete response may compromise the construction of the facets and affect the number of observations in the sample. Table 1 presents the information regarding the treatment and control groups in the pre-intervention period. Year of H. S., Sch. of the Father -Elementary, Sch. of the Father -High School., Sch. of the Father -Declared, Sch. of the Mother -Elementary, Sch. of the Mother -High School, Sch. of the Mother -Declared, Income up to R$ 3.000,00, Higher Income R$ 3.000,00, Cognitive Skill, Access to Cultural Goods 1, Grammar questions 1, Grammar questions 2, Locus of Control and Sociability are statistically different between the treatment and control groups, with at least 90% confidence. This result is expected due to the non-random selection of students to attend the Academia Educar.
According to Table 1, the treatment and control groups are statistically different in the abilities Locus of Control 12 and Sociability. The treatment group presents students with greater emotional instability (Locus of Control) and has a lower degree of sociability. For the other non-cognitive skills, students in the treatment and control group are statistically similar. Table 2 presents the information regarding the treatment and control group before and after the intervention of the program. Note: * 90%, ** 95% and *** 99% confidence.
It is possible to observe that after the intervention the students are statistically different to 90% confidence in all socioemotional skills and political participation. The only variable that decreased value refers to control locus, where lower values mean more emotionally stable students. For all other characteristics the 12 The greater the value of the locus of control, the more unstable the student is in terms of his emotional stability.
values increased, revealing that after the intervention students improve their noncognitive skills and political participation.
It is also noted that the control group before and after intervention are similar in terms of their socioemotional characteristics and political participation. This result is a good indication, as students who did not participate in the Academia Educar program, further reinforcing the results found for the treated students.

Empirical Strategy
The objective of the Academia Educar project is to provide young students with the development of their socioemotional competencies through the protagonism and knowledge of their rights and duties as citizens, so that they are able to transform their reality and their community.
To investigate the causal relationship of these factors several experimental and non-experimental techniques are used in the impact assessment literature. Among them, the method of differences in differences is widely used by researchers, either alone or in combination with other methods. 13 This research uses quasi-experimental techniques to identify the causal relationship of the program to the indicators of interests.
To estimate the causal effect, two control groups are considered, in order to mitigate possible biases of estimation. The first control group consists of students from a school that does not have students belonging to the program, 186 individuals. This control group has the advantage of avoiding the contagion caused by the students of the treatment group, as discussed in section 2. The second control group is formed by students from one of the participating schools, 41 individuals; totaling 227 students. This second control group is used as a way of testing the robustness of the results.
The school not participating in the program chosen for the control group is similar to the schools that participate and is also easier to access to the project management team. The groups that answered the questionnaire were from the same grades that participated in the project and all the students present on the day answered the questionnaire. At the end of the year, the Educar team returned to these same classes.
When selection for participation in a program occurs randomly, this mechanism provides the necessary balancing of observed and unobserved characteristics of the units that make up the two groups, Angrist and Pische (2008). Randomization allows us to create a situation in which there is no correlation between being treated and the attributes of the observation units (participants). If the exposure to 13 The propensity score and the method of differences in differences were proposed by Heckman, Ichumura andTodd (1997, 1998).
the policy is random, one can compare two groups: one that received the program (treatment group) and one that did not (control group). Then, the difference in students' performance in the two situations (treatment and control) is calculated to evaluate the impact of the program. However, due to the non-random selection of the Academia Educar program, this procedure is not the correct one to apply. As shown in Table 2 of section 3, it is noted that the treatment and control groups are distinct in terms of their observable characteristics before the intervention. Due to the significant differences between the treatment and control groups, the propensity score method is used. This method allows finding young people of the control that are as close as possible to the treaties. Pre-project information regarding student demographics, socioeconomic backgrounds (parental schooling, family income, etc.) and cognitive ability (questions of logic applied to the questionnaire) will be used to make these students look alike. 14 In addition, since we collected information on our impact indicators of interest at the beginning of the evaluation, we will also use the difference-differences method. This method allows us to eliminate unobservable differences that are fixed in time. The causal effect of the program can be found statistically. To validate the causal effect, several robustness tests will be performed, since the method used is not without problems.
This strategy allows to estimate the effect of the treatment, that is, the impact of the intervention of the Academia Educar on the social-emotional abilities of the participants of the state and municipal public schools of Campinas / SP. The main hypothesis of the differences in differences method is that the temporal trajectory of the outcome variable for the control group represents what would occur with the treated group, in the absence of the intervention, Menezes-Filho (2012). In our case, it is necessary that the average of the social-emotional abilities have pre--treatment 15 trajectory parallel to that of the participants of the control group. Our specification model is: Socioemocional it in which refers to a socioemotional characteristic of student i at time t; X it are control variables; AE it if participant i is treated, that is participates in the Academia Educar project, and 0, otherwise; t it is a time dummy that assumes the value 1 in the post-program period and 0 in the pre-program period and ε it is the error term of the model. 14 The variables used in the propensity score are: Boy, Age, Cognitive Ability Schooling Father Ens. M. Schooling Mother Ens. M. Student Education 2 Student Education 3 Major Income R $ 3,000.00 Books More than 3 Access to Cultural Goods 2 Access to Cultural Goods 3. 15 Although not a test for this hypothesis, an intuitive way of checking is to test whether the trajectory of the outcome variable for the two groups has the same time trend.
In sum, the hypothesis of causal evaluation of the Academia Educar program on indicators of interest uses: a) the observed variables pretreatment that allows to select from the control group those more similar to those treated; b) the unobserved differences are fixed over time. The combination of these two methods -matching and differences in differences -allows us to identify causality and measure the impact of intervention. Several robustness tests will be performed to validate the results.

Impact Indicators of Interest
As stated in section 2, the objective of the Academia Educar project is to promote the formation of youth leaderships in public schools, creating opportunities for young people to discover their potential and transform their reality (Fundação Educar DPaschoal, [2020]). In short, two measures would satisfactorily address this goal: youth protagonism and citizenship. 16 However, we do not find anything so specific in literature. Therefore, we seek measures correlated with these two aspects. The idea was to use Senna items.
The Senna (Social and Emotional or Non-Cognitive Nationwide Assessment) is an instrument developed by the Ayrton Senna Institute, in partnership with the OECD, which consists of measuring the socioemotional competences of students in the 5th year of elementary school to the last year of high school. The issues that make up the instrument relate to students' attitudes, feelings or perceptions about themselves. It has items to measure the Five Great Personality Factors, the so-called "Big Five": Opening to New Experiences; Extroversion; Kindness; Conscientiousness; and Neuroticism (Emotional Stability), McCrae and John (1992). The items vary in a Likert progressive scale ranging from 1 to 5, ranging from "totally disagree" to "strongly agree," according to a person's belief in having a personality trait or characteristic. In addition, the instrument also has items to measure Locus Control.
For this evaluation, we selected some Senna items. Specifically, we selected items that include the following facets: Sociability and Assertiveness, from the Extroversion construct; Volatility, from the construct Neuroticism; Imaginative, from the construct of Opening to the New; and the items referring to measurement of control locus.
These items were chosen because they are related to the skills that the Academia Educar team seeks to foster in the youth. Protagonism is related to having initiative -to propose ideas; to be able to mobilize, encourage cooperation or teamwork; commitment to goals; leadership. Citizenship is related to knowing their rights and duties -something not contemplated in the items, it is true -but also, 16 School learning can also be an indicator to be considered. But, the option was to focus on indicators related to protagonism and citizenship.
to the idea that 'making it happen' depends on the young person, that is, he must assert his rights, something captured at the locus of control, for example. Thus, the chosen impact indicators of interest are: It is important to clarify that in addition to the Senna items, 'loose' items were also included in the proposed instrument to capture the impact indicators of the Academia Educar. These items were developed by the Academia Educar staff and are in line with the dimensions prioritized by the project. One of the issues included addresses the student's interest in participating in the country's politics. This issue has become one of the indicators of evaluation.
"Do you believe that you should participate in politics in your country?"

( ) No ( ) Yes
The indicators from 1 to 5 were constructed, therefore, from a specific set of Senna 17 items. For this investigation, 3 proposals were used to construct these indicators. The Locus control variable is constructed from 14 items belonging to the Senna questionnaire, since the Sociability feature is constructed from 8 Senna items. The variable Volatility is based on 6 items of the questionnaire, since the Imaginative feature is constructed based on 4 items and Assertiveness is constructed from 2 items of the Senna questionnaire. Finally, the dependent variable Political Participation is a binary variable that assumes the value 1, in case the student believes that he should participate in the policy in his country and 0, otherwise.
Proposition 1 uses 34 items, proposal 2 uses all the items present in the questionnaire, that is, 49 items and proposal 3 is constructed from the items indicated by the factorial analysis. The questionnaire has five levels of responses, thus, for each competency was added the value given for each response and then divided 17 The Likert scale is used in the Senna questionnaire. by the number of items corresponding to each social-emotional characteristic. In this way, each characteristic has a numerical value between 1 and 5.
To measure the internal consistency of these indicators, Table 3 reports the Cronbach's Alpha, whose objective is to evaluate the magnitude in which the items of an instrument are correlated. The internal consistency is better the closer to 1 it is value. It is the reason that we use 3 proposals for building social-emotional skills. Although, there is no precise rule about alpha values, some authors consider the small internal consistency for alphas smaller than 0.21. For alpha between 0.40 and 0.21, the consistency is said to be reasonable. Values between 0.60 and 0.41 are considered moderate. If the alpha is between 0.80 and 0.61, the consistency is considered substantial and for values greater than 0.80, the instrument is almost perfect. According to Table 3, the different proposals present moderate and substantial results, Landis and Koch 18 (1977). The applied diagnostic questionnaire can satisfactorily construct the proposed socioemotional indicators.

The Control Variables
To estimate the impact of the Academia Educar in 2016 on indicators of interest, a large set of characteristics of the students is considered, including age, gender, cognitive abilities, socioeconomic characteristics and family characteristics, as described in Table 2. These variables are used to define if the treatment and control groups are similar -when controlling for these observed variables we are reducing the potential bias from non-observable sources, that is, we seek to find the "clean" treatment effect.
The choice of these characteristics derives from the empirical literature and also from the database informed by Academia Educar. Control variables can be found in Frame 1.

Variables Description
Age Age in years.

Boy
Assumes 1 if student declares males and 0 declares to be female.

Schooling of the father: declared
Assumes 1 if the father's schooling was not quoted and 0 otherwise.

Schooling of the mother: elementary
Assumes 1 if the mother has until the 5th year of Elementary School and 0 otherwise.
Schooling of the mother: high school Assume 1 if the mother has high school and 0 otherwise.

Schooling of the mother: declared
Assumes 1 if the mother's schooling was not mentioned and 0 otherwise.
Income up to R$ 3,000.00 Assumes 1 if the family income is between R$0.00 up to R$3,000.00 and 0 otherwise.
Higher income R$ 3,000.00 Assumes 1 if the family income is higher than R$ 3,000.00 and 0 otherwise.
Cognitive skill Assumes 1 if the student has hit at least one of the two logic questions and 0 otherwise.

Books zero to 3
Assumes 1 if the student reads 0 to 3 books per year and 0 otherwise.
Books more than 3 Assumes 1 if the student reads more than 3 books per year and 0 otherwise Access to cultural goods 1 Assumes 1 if the frequency of cultural goods is "never" and 0 otherwise.
Access to cultural goods 2 Assumes 1 if the frequency of cultural goods is "1 to 6 times a year" and 0 otherwise.
Access to cultural goods 3 Assumes 1 if the frequency of cultural goods is "monthly" and 0 otherwise.
Grammar questions 1 Assume 1 if the student is correct on the grammar question and 0 otherwise.
Grammar questions 2 Assume 1 if the student is correct on the grammar question and 0 otherwise.
Source: Prepared by the authors based on the Educar Academy data source.

Results
In this section, we present the results of the impact estimates for the indicators of interest. In the first subsection, the presented results are based on the model of differences in differences. The subsequent subsection presents the results from a combination of methods using the matching and difference in differences model, which are the preferred results because of the pre-treatment differences already observed between students in both groups.

Estimates of Impact of the Academia Educar Assessment on the Indicators of Interest: Estimation through the Differences in Differences Model
This section presents the results of the diferences in differences model. Table  4 presents the results with two different control groups. Source: Own elaboration based on data from Academia Educar. Note: * 90%, ** 95% and *** 99% confidence.
Control group 1 is made up of students from the Escola Estadual Maria Julieta de Godoi Cartezani and Escola Municipal de Ensino Fundamental Prof. Odila Maia Rocha Brito. Control group 2 is made up of students from the Escola Estadual Maria Julieta de Godoi Cartezani, that is, by students who are in a school that does not participate in the Academia Educar project.
This procedure is adopted, since the treated students can teach to their colleagues who belong to the same school the learning acquired by the Educating Academy, in this way, it is important to divide the two control groups to deal with possible problems of contamination once that it is possible that the school having students in the control group interacting with treated students may lead to an underestimation of project impacts. The results with different control groups is one way of showing the robustness of the results. Table 4 shows that the Academia Educar program had an impact on the Locus competences of control, sociability and political participation. The negative sign of Locus of Control means that the students who participated in the program have improved this competence, i.e., students feel more in control of their own lives, demanding more of themselves and focusing on what they can do on their own to deal with their respective problems. The impact on sociability and political participation are also in the expected direction, i.e., students who participated in the project presented greater variations in the indicators of sociability and political participation than students who did not participate in the project. The results for the variables Assertiveness and Imaginative were positive, but were only significant for control group 1; for control group 2, the impact results decrease in magnitude and are no longer significant. For the variable of interest Volatility, the results were not statistically significant for any of the control groups.

Estimates of Impact of the Academia Educar Assessment on the Indicators of Interest: Estimation through Propensity Score and Differences in Differences Model
This subsection presents the results from a combination of methods, using the propensity score and the difference in difference model. This method allows individuals to be searched in the closest 19 control group and uses the results of these individuals to obtain what would be the outcome of the individual in the treatment group if they were not treated.
First, based on the vector described in Frame 1, we estimated an equation for the probability of the individual participating in the treatment. This procedure was performed separately for each dependent variable analyzed, as well as for each control group. Then, the kernel matching 20 estimates were obtained. In 19 Also called selection in observables, Rosenbaum and Rubim (1983). 20 Each treated student is paired with a weight for each untreated student, so weight declines as the Appendix A are presented the logit models for the probability of the individual being treated (Table 6), as well a balanced quality analysis (Table 7 and Table 8). Table 5 presents the results of the program. Source: Own elaboration based on data from Academia Educar.
According to Table 5, participation in the Academia Educar project has an impact on the variables of interest Sociability, Assertiveness and Political Participation. For Locus control, the result was only significant for the control group 1. Compared to the results in Table 4, the positive results remained for Sociability and Political Participation.
The treated before the intervention were less sociable than the control and after the program the sociability was even greater. This result is expected because of the program's operation, since young people work too hard this skill during the program. Regarding political participation, there was no significant difference distance of the propensity score between treated and untreated students increases. before the program and after the Academy Educate the treaties have a greater political participation. This is also expected because of the character of the program, which seeks to foster the youth protagonism, wants to make them know their rights and duties as a citizen.
Again, no impact was observed on the variable 'Volatility'. For the Imaginative variable, the results indicate null impact for Control Group 1 and negative when we consider Control Group 2. The negative result for Imaginative is not found in other specifications, most of the time the models suggest a positive impact, but statistically insignificant. Table 8, in Appendix A, presents estimates for the same models from Table  4, but using other pairing algorithms -specifically, matching by the nearest neighbor. We worked with 1 and 2 neighbors, always with replacement, that is, the same observation of the control group could be combined with more than one treated unit. As can be observed, the results of Table 4 are confirmed: positive and significant impacts for Sociability, Assertiveness and Political Participation. In these specifications, however, Imaginative appears with a positive and significant signal for control group 1 and not significant for control group 2. No significant and consistent impacts are estimated for Locus of Control nor for Volatility, results also already pointed out in Table 5.
As previously stated, one of the ideas was to explore different ways of constructing the socioemotional measures used in this evaluation. Thus, in Tables 8 and  9 of Appendix A, the same results are shown in Table 5, but using these alternative proposals for measures of social-emotional indicators. In these tables of the appendix no results are presented for the variable of political participation since it is derived from a single question of the questionnaire, therefore, there are no different possibilities of measurement for this variable.
The results of Tables 8 and 9 confirm the impact of the Program on the measure of sociability -in all specifications the results are positive and significant, indicating that Educar contributes to a greater sociability of its participants. For other measures, the results are as strong. The Locus of Control, Imaginative, and Assertiveness variables present positive and significant results in some specifications, but in general it does not appear that the results are robust. For the variable Volatility, as in the first specification, the results suggest that there is no impact.
In terms of magnitude, according to the data in Table 2, the mean of the Sociability variable for the pre-intervention treaties was 3.127 -this means that an impact of 0.396 (according to the results in Table 5 -control group 2) represents 12% of the value of the variable in the baseline. The same rationale for the Assertiveness variable implies a 16% impact of the value of the variable at the baseline (mean before treatment was 2.92 with impact of 0.463). For the variable of interest in participating in the country's policy, the impact is quite large: impact of 42 percentage points.
At the baseline 47% of the students said they had an interest in participating in the country's politics; due to the participation in Projeto Educar Project, in the end, 90% of the students said they are interested in participating in the country's politics.

Final Considerations
The objective of this study was to present an impact assessment of the project Academia Educar of Educar DPaschoal Foundation, specifically the final results of the impact assessment on aspects related to socioemotional skills of students. Two field surveys were carried out to carry out the evaluation. A first cadastral survey of each student in early 2016; and a second survey at the end of the same year. The two surveys collected the impact indicators of interest, related to socioemotional characteristics.
The evaluation was based on the treaties that participated in the program in 2016. The control group chosen was formed by students who did not participate in the Academia Educar project. There are two distinct groups of controls, the first group is made up of students from two partner schools of the DPaschoal Foundation, but one of the schools has young people participating in the project. The second control group is formed only by young people from the school that is not participating in the program. The hypothesis of identification for the causal evaluation of the Academia Educar program on socioemotional indicators and political participation utilizes the observed pre-treatment variables as well as the method of differences in differences to deal with unobserved variables that are fixed in time.
The results of this evaluation should be read with all the necessary caution when we think that we do not have the perfect instruments to measure social skills, as well as when we remember that we did not have a lot to define the treaties, but rather a selection process that seeks to choose the most motivated / interested to participate.
We use different impact indicators associated with the socioemotional skills of the project beneficiaries. According to the results obtained, the program has a positive and statistically significant impact on Sociability and Political Participation. For assertiveness we also defend that there is an impact. The results obtained for this variable in proposal 1 are positive and significant for all specifications that use the combination of matching and differences in differences. Therefore, we conclude that this variable also has an impact. The results for Locus Control and Imaginative variables were significant and in the correct direction in part of the specifications; for Volatility, the results do not suggest impact.
It is important to emphasize, the evaluation relied on baseline data and with several robustness exercises, which leaves us more confident in terms of the credibility of the results. One suggestion would be to accompany the students, so that long-term effects could be measured and it would be possible to know the effectiveness of the Academia Educar, that is, the duration of the impact.  Source: Own elaboration based on data from Academia Educar. Note: Mean differences are statistically different from zero with up to 95% confidence. Source: Own elaboration based on data from Academia Educar.