To substantiate our achievements and strengthen our work, Inside Out employs multiple evaluation methods overseen by the Artistic Director:

1. Weekly Artist Leader meetings to assess each child’s progress artistically & personally

2. Pre/post participant evaluation surveys

2008/2009 The School Project ACT 2 results

2009/2010 The School Project ACT 1 results

3. Parent questionnaires distributed in performance programs

4. School administrator & faculty feedback during bi-monthly site visits

5. Participants’ feedback at closure workshops

6. Post-program assessment meetings with Artist Leaders

7. An annual artists’ retreat that allows artists to assess the program

 

Dr. James S. Catterall's groundbreaking programming evaluation of Inside Out's School Project as published in Teaching Theater magazine, Winter 2006

Download a .pdf version of this article

 

Inside Out's School Project

A research report measuring the power of a theatre-based program for at-risk junior high students

By James M. Catterall (Photos by Jennifer Lyn Browne)

Ready for the play: an Inside Out student at the 2005 School Project campout.

THIS ARTICLE REPORTS on the study, An After-School Program in Thematic Theatre for At-Risk Middle School Students. The focus of the study was the School Project, a program sponsored and implemented by Inside Out Community Arts in Venice, California that serves about one hundred public middle school students at three Los Angeles area middle schools. The broad goal of the program is to promote healthy interaction and development among middle school students at risk of school failure and engaging in destructive and anti-social behaviors. More specifically, the School Project’s activities were aimed at increasing students’ motivation and skills for conflict resolution through specific drama activities.

As researchers, we took special interest in this program for several reasons. One was that the project is consistent with a developing body of research showing that drama and theatre can promote specific learning and development. Another is that there are few formal studies focusing on the impact of theatre on adolescents. Another point of interest was the program’s af-ter-school format, a model of substantial interest to arts educators as the regular school day is increasingly directed to basic skills, including the testing of math and reading under the No Child Left Behind Act. And finally, the School Project involved work with kids over an ample period of time, thus offering reasons to believe that important developments could result. Too many studies of learning and development take place over a matter of a few weeks or less, rendering findings as mere suggestions.

This is not to say that there isn’t a theoretical foundation for the program or for this research. One visible study in playwriting showed effects on student self-confidence, writing ability, and persistence in school (Horn, 1992; for detailed publication information on this and other studies and reports referenced here, see the resource list on page 8). Additionally, after-school programs in the arts have shown specific benefits aligned with the goals of the School Project. Stanford University’s Shirley Bryce Heath (1999), an anthropologist who studied three after-school programs—including those that featured theatre—for a decade, found that arts programs offer unique opportunities for youth to take on and develop leadership roles—an experience that participants rarely encounter in regular school settings.

Much of the School Project is about developing a more positive sense of self, strength in standing up for one’s own views and ideas, respect for the views of others, and more positive social interactions—all things that are potential outcomes of collaborative creation of dramatic works. The curriculum is also consistent with the California Visual and Performing Arts Standards for Theatre, which stress acting, oral communication, and creation and invention in theatre. The program we assessed used theatre, writing, movement, drawing, voice, and visual arts exercises as building blocks for students to write and perform original plays.

Ninety-minute workshops took place after school once a week at the three middle schools. At each of the three sites there were six specially trained professional teaching artists including actors, writers, directors, dancers, singers, and visual artists. In addition, a teacher from each site acted as a liaison between the program and the school and participated in the workshops.

During the program’s first half, about twelve weeks, at each school the artists led their three groups of roughly eleven students through a course of exercises that introduced students to the elements of creating a play (character development, story, and structure), while simultaneously promoting teamwork and strengthening socialization, communication, and conflict resolution skills. All exercises involved student interaction and were structured so that every child participated. The curriculum enlists engaging exercises, such as theatre games, creative writing, improvisation, puppetry, performance art, group drawing, and dance as a platform for experiential learning.

The second half of the program was devoted to student creation of plays and to performances.

What follows is a report of our study of the School Project carried out at the three middle schools during the winter and spring of 2001.

Study design

We built on an earlier informal study of the School Project that served to define what the sponsors were interested in, but that had not mounted an evaluation design that could say much about the program’s influences on actual participant development. In order to attain valid inferences about the impacts of the program, we identified an ideal comparison group and also worked with the program to sharpen the sponsors’ sense of the big ideas they were after. Most significantly, our design enlisted a comparison group that had signed on to the program—students who had self-selected into the program but who, because of space limitations, could not be accommodated. This established a robust control or comparison group for the study. We also took the primary questions included in a previous program survey and clustered groups of questions around the main ideas they represented. So we ended up creating multiple-item scales to measure key ideas. This is a common practice in psychological studies probing learning and motivation.

The student population. Three middle-school sites in the Los Angeles area hosted this study: a South Central Los Angeles school (42 percent Latino, 57 percent African American, and 3 percent other), a northwest Los Angeles school (51 percent Latino, 27 percent white, 9 percent African American, 8 percent Asian, and 5 percent Filipino), and a northeast Los Angeles school (76 percent Latino, 22 percent Asian, and 2 percent other). More than 80 percent of the School Project participants were from low-income families, and two of the three school sites involved in this study (northeast and South Central) are among the lowest academic performing schools among the Los Angeles Unified School District. Participants came from widely differing cultures and had a range of behavioral, academic, and language difficulties. The students ranged in age from eleven to fourteen.

Study goals. This study was designed to gauge several of the critical social and artistic developments targeted by the program. The principal means of generating and gathering data was through student surveys administered to all students prior to the program and again after completion of the program (this is a pre-test, post-test design). The items for the survey were generated from questions used in a previous study of the School Project. The items in the survey instrument were reorganized conceptually and supplemented using questions based on standard instruments designed to assess student attitudes and motivation (questions and scales developed and used by Eccles, 1999 and Catterall, 1994). The instrument was built around nine key constructs for which multiple items could help measure student development in several important broad areas. (The item “Attitudes about acting” was based on a single question about whether or not a student liked to act.)

These key measurement areas included:

• Ability to work with others with whom a student disagrees.

• Ability to work with others in groups generally

• Problem resolution skills.

• Sense of self-efficacy (the degree to which a student feels his/her actions make things happen).

• Awareness of one’s own thinkingprocesses (metacognition).

• Attitudes about doing arts.

• Attitudes about acting.

• General outlook for the future.

• General satisfaction, happiness.

The study was conducted through a treatment comparison design in which seventy-one program students and eighty-four non-participating comparison group students completed questionnaires before and after the students engaged in the program. The non-participating students did not engage in School Project activities over the time of the study.

Specific conditions built into this study served to raise the validity and reliability of the generalized conclusions beyond what pre-post designs typically manage to do. One contributor was that our assessments of changes in students’ perceptions of skills, attitudes, and motivation were based not simply on changes in self-reported pre- and post-measures. The changes reported for participating students were set against parallel pre- and post-assessments of students not involved in the program as a basis for comparison. Moreover, the comparison students came from the same schools as the subjects. And most important, the comparison student group was composed of students who had signed up for or who had signaled interest in the program, but who could not participate because the program lacked sufficient space. This averts a problem that hinders many treatment-control designs— namely the self-selection bias inherent in measuring effects on students who are probably more motivated to participate than comparison students.

All students expressing an interest in the program at the start—those eventually participating as well as those who had to wait—were given the pre-survey. School psychologists and teachers assisted in the administration of the surveys and also were instrumental in rounding up students, especially students in the comparison group, for the post-assessment after the end of the program.

Data analysis methods

We used two related perspectives in analyzing our data. Because the study enlisted a valid control group, the main analysis focused on the difference in scale change scores pre- to post- for program students versus comparison students. Since the measures favored program students across the board, we called these “net scale gains” for School Project students.

We also examined the absolute pre- to post- scale score changes for program students and evaluated these changes in what statisticians refer to as effect sizes. An effect size is calculated by dividing each net scale score change by the standard deviation of the comparison group’s pre-score on each scale. This implies that if a comparison group member’s pre-scores on a scale such as self-efficacy are concentrated around a particular value, it is impressive if program students average a post-score that is significantly higher than the pre-score. The difference is then measured in standard deviation units.

This is an appropriate statistical approach designed to gauge the magnitude of score changes in a way that allows comparisons across diverse scales—scales for which one-point differences can mean differing degrees of change in the underlying attitudes or opinions. As a simple example, it is problematic to compare a one-point change on a five-point self-efficacy scale to a one-point change on a five-point scale assessing interest in arts. The use of effect sizes serves to standardize both scales and allows comparing levels of impact in a common unit of measure.

Net scale gains for School Project students

Our main analysis compared program and control student scores by examining the net changes in program versus comparison student scores for each of the nine scales. For example, if program students averaged a gain of 0.5 on a scale while control students averaged a gain of 0.1 on the same scale, the net gain for the program group would be 0.4. The results of these comparisons were held up to a traditional test of significance for the difference in the two groups’ average gains. (The significance values shown in the table below are based on t-tests for differences in program students’ versus comparison students’ average score changes.)

The scales, the net gains, and their statistically significant p-values are shown in the first three columns of the table. As the table illustrates, participating School Project students made meaningful and significant gains on seven out of nine scales. The largest net gains were in student attitudes about doing arts, acting, metacognition, problem resolution skills, and self-efficacy. Small but statistically significant gains were also shown for students’ perceived ability to work in groups and in a general life-satisfaction measure. In the other two scales, the net gain in outlook for the future bordered on significance (p < .07) and the small net gain in student perceptions of their abilities to work with others whose ideas differ was not statistically significant. This last measure illustrates a feature of the net gain framework: both program and comparison students showed gains in this “working with differences” scale. Thus the net gain for School Project students was relatively small.

Scale gains and effect sizes

For a second way of perceiving and interpreting our data, we asked just how much program students gained (or lost) on each of the nine scales and how meaningful were the changes, independent of how comparison students scored. As shown in columns 5 and 6 of the table, students showed absolute gains on eight of the nine scales. The relative strength of these changes is expressed in effect sizes shown in column 6. This perspective brings some reordering of effects, but a general picture remains intact. Most student perceptions of important personal and interpersonal skills registered high effect in both views: these include metacognition, problem resolution skills, working well in groups, and self-efficacy. Self-efficacy showed the largest effect in both analyses.

Absolute gains in the first four scales of the table translate into relatively small effect sizes. This is mainly because the absolute gains made by program students were smaller than the net gains when accounting for control group scores. (These net gains are shown in column 2.) The student life satisfaction and outlook scales illustrate the relationships between the two measures. Program students showed no gain (actually a very small decline) on the life satisfaction scale overall. But the net gain was appreciable. The explanation for the difference is that the comparison studentsshowed a loss on this scale pre topost. This difference is potentiallyimportant. Studies using motivationrelated indicators over time within a school year often show declines between fall and spring measures forthe same children. Thus a program may succeed simply by sustaining any relatively high beginning student motivation levels over time. Our data revealed modest program student gains and high comparison group declines for measures of attitudes about art and acting as well as for general life satisfaction.

Student reports of learning on post-surveys

Our results in the program-control scale change analyses were consistent with what the participants volunteered they had learned on the post-survey, an added source of assessment information. In this survey, students highlighted what they learned. A content analysis of their comments showed that student views centered on the following areas of program effect:
• Expressing feelings
• Self-efficacy
• Teamwork and cooperation
• Respect
• Self-confidence
• Controlling emotions
• Imagination
• Friendship
• Acting skills
These expressions were consistent with scale-based findings about group skills, working with others, dealing with interpersonal differences, reflective thought, self-efficacy, and attitudes about doing arts. The post-survey also included a section of sixteen multiple choice and open response questions that gave participating students a chance to comment on their School Project experience. The questions covered several areas including the students’ feelings towards program artists, their reasons for signing up for the project, and what they learned as a result of their participation. The students’ answers to these more informal queries paralleled those they offered to the scaled questions used in our analysis of the program’s effectiveness. As a second way of perceiving and interpreting our data, we asked just how much program students gained (or lost) on each of the nine scales and how meaningful were the changes, independent of how comparison students scored. As shown in columns 5 and 6 of the table, students showed absolute gains on eight of the nine scales. The relative strength of these changes is expressed in effect sizes shown in column 6. This perspective brings some reordering of effects, but a general picture remains intact. Most student perceptions of important personal and interpersonal skills registered high effect in both views: these include metacognition, problem resolution skills, working well in groups, and self-efficacy. Self-efficacy showed the largest effect in both analyses.

Theoretical considerations

The effects we have reported here are consistent with findings from previous research. And as a cluster of effects,the findings of this study are also theoretically coherent. Recent studies of programs where students or children gain skills in artistic expression point to self-efficacy as an important outcome. (For examples see J. S. Catterall and K. A. Peppler, Arts Education and the World-views of Inner City Children, The FordFoundation, 2004.) The most straight-forward explanation for why this occurs is that learning art, drama, ordance is itself empowering. Feelingsof competence and control come with added expertise in the arts. In the case of the School Project, the publicnature of artistic skill development—learning to act and perform exercises with peers—reinforced self-perception in ways that more private learning could not, at least over relatively shortperiods of time such as the several months of this program. As we have found in this study, achieving competency in theatre and increased facility with social and moral issues or dilemmas are likely to boost a student’s ability to work in groups, solve problems, and work with others whose ideas differ. And the School Project encourages reflective thought, or metacognition, in the eyes of the students—practices considered likely to accompany artistic creation. Overall, the cluster of main effects showing in the scales seems very consistent with the program experienced by the participating students.

Set panels created by School Project students for their plays.

Discussion

The School Project appears to have had a significant positive influence in the lives of the middle school students it served during the course of this study. The clientele are in need of social support and are at risk of school failure and destructive and antisocial behavior. Many of the participating children had never experienced a sustained, purposeful after-school program such as this effort.

The centerpiece of this analysis was the measurement of important attitudes and self-perceptions of participating students before and after they took part in the program. These measures were set against a rigorous comparison group—a group consisting of students who had signed on to the same program but for whom there was insufficient space.

Program students showed significant gains in most measured domains: sense of self-efficacy, working in groups, problem resolution skills, and awareness of their own thinking skills. The domain working with others with whom they disagreed showed a modest but meaningful effect size. The effect sizes across the five areas ranged from moderate to strong—from +0.28 to +0.62 standard deviations. Gains in student attitudes about doing arts activities were modest when measured in effect size and dramatic when assessed as net gains in comparison to control student scores. Two more scales—outlook for the future and life satisfaction—showed positively but the differences were small.

Finally, based on students’ written comments, the School Project appears to be a program where the older children (eighth graders) worked positively with the younger ones (sixth and seventh graders). Participants expressed sentiments that they would like to serve as program mentors in the future. As such, it seems that in ways anticipated by sponsors, this program is a training ground for adolescents who could volunteer to take part in future editions of the School Project or through other avenues.

James S. Catterall is professor of education at the Graduate School of Education and Information Studies at the University of California, Los Angeles. He was a contributing author of Champions of Change: The Arts and Human Development (1999) where he reported studies of general involvement in the arts as well as instrumental music. He also was a principal contributor to Critical Links: Learning in the Arts and Student Academic and Social Development (2002), a compendium of research studies, in which he wrote the section on learning in drama as well as the volume’s general analysis of the transfer of learning from the arts.

Participant Survey:

Have you attended INSIDE OUT before? Y / N (Circle One)
Date_______________________
M / F (Circle one)
Inside Out
Participant Profile and Checklist
Instructions:
SURVEYS MUST BE FILLED OUT IN BLACK OR BLUE PEN!!!!!
Please circle the response which best describes your opinion about the following items.
There are no right or wrong answers. If you have questions, please ask your teacher for help

(SA) STRONGLY AGREE

(A) AGREE

(NAD) NEITHER/AGREE or DISAGREE

(D) DISAGREE

(SD) STRONGLY DISAGREE

1
I look forward to facing and working to overcome challenges.
SA A NAD D SD
2
When I need help, I find someone to talk with.
SA A NAD D SD
3
I feel confused a lot of the time.
SA A NAD D SD
4
Most disagreements lead to fights.
SA A NAD D SD
5
I feel like I am all alone in the world.
SA A NAD D SD
6
Talking about my feelings with friends helps me solve my problems.
SA A NAD D SD
7
When I disagree with someone, it means I don't like them.
SA A NAD D SD
8
Doing creative activities is not important.
SA A NAD D SD
9
I don't like to work on projects with a group of people.
SA A NAD D SD
10
I am patient about getting what I want.
SA A NAD D SD
11
Other people listen to me when I express my ideas.
SA A NAD D SD
12
I tend to get angry when anyone disagrees with me.
SA A NAD D SD
13
Being involved in Inside Out is for other kids, not me.
SA A NAD D SD
14
Disagreeing with someone is the same as being angry with them.
SA A NAD D SD
15
I need to be in control most of the time.
SA A NAD D SD
16
I feel scared about my future.
SA A NAD D SD
17
Talking about my feelings with adults helps me solve my problems.
SA A NAD D SD
18
I feel respected by my fellow students.
SA A NAD D SD

I like to do the following:

19
Read stories
SA A NAD D SD
20
Participate in school activities
SA A NAD D SD
21
Creative writing
SA A NAD D SD
22
Draw or Paint
SA A NAD D SD
23
Talk in front of groups
SA A NAD D SD
24
Act or perform
SA A NAD D SD
25
Spend time in nature
SA A NAD D SD


(SA) STRONGLY AGREE

(A) AGREE

(NAD) NEITHER/AGREE or DISAGREE

(D) DISAGREE

(SD) STRONGLY DISAGREE

Remembering the past four months, I think about:

26
How happy I have been
SA A NAD D SD
27
How much I have learned in school
SA A NAD D SD
28
The new friends I have made
SA A NAD D SD
29
How much I have learned about myself
SA A NAD D SD
30
How often I have made my own decisions
SA A NAD D SD
31
How I have been able to solve my problems with help from others
SA A NAD D SD
32
How much I look forward to the future
SA A NAD D SD
33
How I have changed my way of thinking
SA A NAD D SD
34
How I have found new ways to deal with my problems
SA A NAD D SD
35
How I have more choices today than I had four months ago.
SA A NAD D SD


(SA) STRONGLY AGREE

(A) AGREE

(NAD) NEITHER/AGREE or DISAGREE

(D) DISAGREE

(SD) STRONGLY DISAGREE

36
Graduating from High School
SA A NAD D SD
37
Going to College
SA A NAD D SD
38
Having many friends
SA A NAD D SD
39
Being in control of my own life
SA A NAD D SD
40
Quitting school
SA A NAD D SD
41
Enjoying my life
SA A NAD D SD
42
Being someone others look up to
SA A NAD D SD
43
It is hard to avoid getting into physical fights.
SA A NAD D SD
44
It is hard to express my feelings to others.
SA A NAD D SD
45
It is hard to express thoughts or ideas to others.
SA A NAD D SD
46
I am uncomfortable being alone.
SA A NAD D SD
47
It is hard to hold on to my opinions when talking to my friends.
SA A NAD D SD
48
It is hard to have a good time.
SA A NAD D SD

On a scale of 1 to 10 (10 meaning you are very good and 1 meaning you are not so good),
please rate how good you are at the following:

1 2 3 4 5 6 7 8 9 10
not so good > > very good

49
Working with someone who has different opinions than mine.
1 2 3 4 5 6 7 8 9 10
50
Setting goals and working to achieve them.
1 2 3 4 5 6 7 8 9 10
51
Expressing my feelings in a group.
1 2 3 4 5 6 7 8 9 10
52
Being a leader in a group.
1 2 3 4 5 6 7 8 9 10
53
Listening when someone else talks about their problems.
1 2 3 4 5 6 7 8 9 10
54
Figuring out solutions to my problems.
1 2 3 4 5 6 7 8 9 10
55
Making my own decisions even if others don't approve.
1 2 3 4 5 6 7 8 9 10
56
Listening to someone when their opinion is different than mine.
1 2 3 4 5 6 7 8 9 10
57
Finding friends with whom I have many things in common.
1 2 3 4 5 6 7 8 9 10
58
What was the most difficult thing you did in Inside Out (other than the hike)?

59
How would you best describe your relationship with the Inside Out artists? (circle 1)


a. Teacher/student


b. Friends


c. Adult/child


d. Peers


e. Leader/follower


f. Team mates


g. Mentor


h. Other___________________

60
How would you best describe your relationship with the Inside Out teacher from your school?


(circle one)


a. Teacher/student


b. Friends


c. Adult/child


d. Peers


e. Leader/follower


f. Team mates


g. Mentor


h. Other___________________

61
How would you best describe your relationship with other Inside Out participants?


(circle one)


a. Team mates


b. Friends


c. Enemies


d. Competitors


e. Equals


f. Other_____________________

62
What made you decide to take part in this program?

63
Why did you keep attending?

64
Would you enroll in this program again?
Yes No

65
Are you comfortable with the Inside Out leaders?
Yes No

66
Did you get to know the Inside Out artists more or less than other adults in your life such as teachers, counselors, coaches?

67
Besides having fun, what do you appreciate most about Inside Out?

68
What would you change about this program if you could? Why?

69
What do you feel you learned from this program?

70
What do you feel you learned about yourself from this program?

71
How did you feel about your group’s play?

72
How did you feel the night of the performance, before and after the show.

73
Any other comments?



Appendix A Appendix B

(These appendices are PDF files, which can be viewed using Adobe Reader)