Section 5: Outcomes and Benefits from Participation
One of the goals of our research was to identify outcomes and benefits of participation in communities of transformation (CoTs) as perceived by community members. This was in line with our second research question: What are the perceived benefits of participation in a STEM reform community of practice or network for the individual participants and for their campuses? In this section, we describe the outcomes we identified through our research, as well as some trends in the data regarding these outcomes. Later, in section 6, we look in more depth at the ways that engagement and community design are associated with these outcomes.
We asked participants to assess the extent to which their involvement in their communities contributed to meeting 26 individual-oriented benefits and 13 outcomes related to their organizations. Based on the literature on communities of practice (CoPs), we had an idea of the kinds of outcomes that might be met through involvement in the four communities of transformation. The outcomes we settled on were informed by this literature, as well as by qualitative data gathered through interviews and observations. As you can see in Tables 5.1 and 5.2, we performed exploratory factor analysis on these items and uncovered five broader constructs on which 30 items could be grouped. Three of these outcomes relate to the individual, including learning and improving teaching practice, skills for leadership and change, and networking, while the other two pertain to departmental and institutional outcomes. The remaining items that did not load on a construct represent other important outcomes that contribute to STEM reform (see Table 5.3).7
Table 5.1: Scale Scores and Factor Loadings for Individual Outcomes Scales
|Individual Outcome – Scalesa||Factor Loading||Cronbach α||M (SD)b|
|Learning and Improving Practice||0.95||3.33 (1.05)|
|-Led to changes in teaching practice||0.90|
|-Motivated me to be innovative in practice||0.89|
|-Led to professional growth to improve practice||0.82|
|-Gained access to new curricular/pedagogical resources||0.79|
|-Contributed to intellectual growth||0.77|
|-Provided examples to model work after||0.68|
|-Contributed to understanding big picture of STEM reform||0.57|
|d-Gained skills to overcome barriers to change||0.85|
|-Empowered to influence change on campus||0.76|
|-Helped develop skills to be a leader||0.72|
|-Motivated me to overcome barriers at home institution||0.72|
|-Gained ideas for contributing to change on campus||0.60|
|-Expanded personal support network||0.87|
|-Connected to people who share personal interests||0.86|
|-Connected to people who share professional interests||0.77|
|-Expanded professional support network||0.73|
|-Connected to a local (i.e, geographic proximity) network||0.67|
|NOTE:aFactor loadings and Cronbach α’s calculated for total sample only. bFive-point scale with 1=Not at all, 3=To some extent, and 5=To a great extent.
Table 5.2: Scale Scores and Factor Loadings for Organizational Outcomes Scales
|Organizational Outcomes – Scalesa||Factor Loading||Cronbach α||M (SD)b|
|Departmental Change||0.92||2.39 (1.09)|
|-Led to changes in teaching practice in my department||0.90|
|-Led to curricular changes in my department||0.93|
|-Led to changes in educational values in my department||0.90|
|-Informed departmental strategic planning||0.80|
|Learning and Improving Practice||0.95||3.33 (1.05)|
|-Led to changes in teaching practice||0.90|
|-Led to developing communities of practice at institution||0.96|
|-Led to developing campus network for STEM reform||0.95|
|-Led to curricular changes in other departments||0.91|
|-Led to emergence of new campus leaders for change||0.82|
|-Led to changes in teaching in other departments||0.80|
|-Led to changes in educational values in other departments||0.74|
|-Informed campus strategic planning||0.68|
|-Led to campus workshops and professional development||0.66|
|NOTE: aFactor loadings and Cronbach α’s calculated for total sample only. bFive-point scale with 1=Not at all, 3=To some extent,
and 5=To a great extent.>-Led to campus workshops and professional development
Table 5.3: Scores for Individual Outcomes
|Individual Outcomes – Individual Items||M (SD)a|
|Recharged and/or energized in work||3.27 (1.25)|
|Allowed to have fun in professional environment||3.10 (1.28)|
|Lent credibility for approach to teaching||3.01 (1.33)|
|Lent credibility for approach to professional work||2.84 (1.34)|
|Afforded opportunities to collaborate on projects||2.58 (1.35)|
|Assisted in career advancement||2.30 (1.30)|
|Gave opportunity to pursue new grants or major projects||2.28 (1.33)|
|Led to publications||1.72 (1.15)|
|Gave skills to make transition from faculty to administration||1.88 (1.26)|
|NOTE: aFive-point scale with 1=Not at all, 3=To some extent, and 5=To a great extent.|
Figure 5.1 shows the frequency of benefits through community involvement cited by the participants. In general, our participants reported the greatest benefits of involvement in these communities coming in the form of learning and improving their practice, reenergizing them in their satisfaction and fulfillment in their work, and gaining credibility for their work related to STEM reform. These individual benefits are important for different reasons among faculty engaging in STEM reform, and we delve deeper into these outcomes below.
Learning and Improving Practice
Of the three broad outcomes we mentioned above, learning and improving practice was the most frequently cited benefit of community involvement (67.3%). This is encouraging for the work of STEM reformers, as more than two in three community members indicated that they changed their teaching and improved in practice as a result of their involvement in these communities. Much of the work of these communities is focuses on pedagogical reform, which comes in the form of applying specific methods in the classroom (such as process-oriented, guided-inquiry learning) or adopting a new perspective or approach to science education (such as studying science through complex issues like HIV and climate change). When we asked participants in our interviews to identify the most important aspect of their community involvement, faculty would often cite the pedagogical approach to science education in their community. As one of the primary foci of STEM reform remains getting faculty to adopt evidence-based teaching practices (Dancy & Henderson, 2008), this finding points to the value of these communities in these efforts. However, effective STEM reform requires more than simply engaging faculty in improving teaching; the other frequently cited benefits in this study highlight other ways in which these communities contribute to STEM reform.
Reenergizing Faculty in their Work
Feeling reenergized and having fun in a professional environment are the two most frequently cited benefits of community involvement (73.7% and 69.2%, respectively). STEM reform can be difficult work, especially for faculty seeking to change their teaching practice. They face many barriers to making meaningful changes, including institutional reward structures that undervalue innovation in teaching, disciplinary cultures that place more emphasis on research, and a lack of institutional leadership that fosters a culture that values teaching (Austin, 2011; Henderson, Beach, & Finkelstein, 2011). Faculty who value teaching and are interested in student learning may reach a point where they lack the energy or motivation to keep pursuing this work in the face of these barriers. Our interviews and observations revealed that faculty felt renewed in their efforts due to the supportive and innovative spaces of these communities. Many spoke of the difficulties of engaging in reform in institutions that were less supportive, and they were able not only to develop or regain a passion for this work, but they also looked forward to the chance to let loose and have fun. The role these communities play in helping faculty tap into this passion and sense of fun is vital to their success.
Lending Credibility to Faculty and their Reform Work
Faculty participants also indicated that they gained credibility for their approaches to teaching (64.7%) and professional work (59.5%) due to their community involvement. As we mentioned above, institutional and departmental structures in which STEM faculty often find themselves can tend to focus on research over teaching and student outcomes. Faculty who are interested in such educational aspects of their work may feel that they are seen as less credible due to these interests. The communities in our study work hard to engage faculty members in this meaningful work, and they also are attuned to the structures that can inhibit faculty in these pursuits. Leaders of these communities play key roles in supporting faculty members who are actively involved, even going so far as writing letters of support for tenure and promotions and communicating with community members’ institutions about their accomplishments and how their work contributes to broader progress in STEM reform. These communities also serve as the primary home for professional development for many faculty in our study, and the nature of the work and value communicated by the leadership of these communities go a long way in supporting faculty in pursing these efforts.
While less prevalent, we found that participants also indicated some organizational benefits from their involvement in these communities. Nearly 35% of participants indicated that engagement in these communities contributed to some extent to changes related to STEM reform in their departments, while more than one in five of the participants indicated that some sort of institutional change had come about as a result of their involvement in these communities. We interviewed several individuals from a variety of institutions who were able to infuse the practices from one of the communities into their departments on campus. This often occurred when multiple members from an institution were engaged in the community, especially when this included administrative leadership (e.g., department chairs, deans). We found that departments and even small institutions or schools of science utilized the principles of the communities to guide curricular and strategic planning for their science education efforts. The United States Military Academy at West Point and Brigham Young University Idaho are two good examples of institutions that utilized the SENCER Ideals (reproduced in Appendix 4A) as a guide for science courses throughout their curricula.
We have seen how the broad aggregate results of our research point to the benefits that communities of transformation can provide to STEM reform. We will now highlight several findings from our multivariate analyses that suggest particular reasons why administrators and faculty might gain from engagement with communities of transformation such as the ones in our study.
Multiple Benefits of Ongoing Involvement
For faculty members who pursue ongoing support for teaching and professional development, these communities of transformation offer a highly meaningful experience. Our analyses showed that faculty who were more continuously involved in these communities (i.e., attended more events and engaged more frequently with the community) exhibited greater benefits related to learning to improve practice, leadership skills, and networking (See Table 6.3). Our interviews and observations revealed that faculty who continuously engaged with these communities enjoyed ongoing support from the leadership of the CoTs and often reconnected with their own passion for teaching and for their disciplinary work.
This dovetails with the fact that feeling reenergized in one’s work was the most prevalent benefit cited by faculty in our surveys.
The benefits from community involvement we examined—leadership skills, improving practice, networking, grant opportunities, opportunities for career advancement, and others—are likely attractive to many faculty members. When we examined motivations for faculty to become involved in these communities, we identified some important patterns. Faculty in our study cited a host of reasons that motivated them to join these communities. While some of these faculty were oriented toward a specific kind of benefit, such as the desire to improve teaching, we found that participants benefited in several outcomes even when they were not seeking those particular benefits. For example, faculty seeking to improve their teaching practices (the most oft-cited motivation for involvement) not only reported greater benefits in learning and improving practice, but also reported greater benefits relating to leadership development and networking. In fact, higher scores for all five of our broad benefit constructs resulted among participants who identified any of the following motivations for involvement: support for change on campus, strategies to involve peers in change, opportunities for grant funding, and connecting to like-minded colleagues. This suggests that participation can lead to multiple benefits beyond one’s initial reasons for engaging in these communities.
Our qualitative interviews provided a good example of these expanding benefits of involvement. One faculty member described entering a community initially because she was interested in improving an introductory course, but through her involvement she began to engage colleagues in rethinking the departmental curriculum. As a result of her passion for improving teaching and her knowledge gained through involvement in the community, she moved into a role in the Center for Teaching and Learning. The initial benefits pushed her into new and more effective areas over time. This story is an example of a common narrative arc within our interviews, which was mirrored broadly in our survey results.
Who Benefits? Gains for Women and Faculty of Color
By comparing reported benefits according to two characteristics particularly important to STEM reform—gender and race/ethnicity—we uncovered some significant trends (see Table 5.4). For nearly all of the individual benefits we studied (except for involvement leading to publications), female faculty members reported statistically significantly greater benefits resulting from their participation in these communities as compared to their male counterparts. While all such disproportionate benefits are notable, we are especially drawn to the difference between men and women reporting improvement in skills for leadership and change, gaining skills to transition from faculty to administration, and contributing to career advancement. In a similar fashion, faculty of color indicated greater benefits than White faculty members in several key benefits—networking, being afforded the opportunity to pursue new grants or projects, gaining the opportunity to collaborate with others on projects, gaining credibility for their approach to professional work and teaching, and gaining skills to make the transition from faculty work to administration—as a result of their involvement with these communities.
Table 5.4: Outcome Variable Mean Comparisons by Gender and Race/Ethnicity
|Individual Outcomes – Individual Items||Female
|t Statistic||Effect Size
|Learning and Improving Practice Scale||3.47 (1.00)||3.18 (1.08)||6.50***||0.28|
|Leadership/Change Scale||2.72 (1.14)||2.53 (1.11)||3.61***||0.16|
|Networking||2.64 (1.09)||2.43 (1.12)||4.35***||0.19|
|Departmental Change||2.44 (1.09)||2.34 (1.10)||2.07*||0.09|
|Institutional Change||2.08 (1.09)||2.02 (0.96)||1.16|
|Recharged and/or energized in work||3.45 (1.20)||3.05 (1.27)||7.78***||0.33|
|Allowed to have fun in professional environment||3.26 (1.24)||2.90 (1.30)||6.61***||0.28|
|Lent credibility for approach to teaching||3.16 (1.32)||2.83 (1.32)/td>||5.81***||0.25|
|Lent credibility for approach to professional work||3.00 (1.32)||2.67 (1.33)||5.74***||0.25|
|Afforded opportunities to collaborate on projects||2.68 (1.36)||2.48 (1.32)||3.52***||0.15|
|Assisted in career advancement||2.39 (1.34)||2.20 (1.25)||3.46**||0.15|
|Gave opportunity to pursue new grants or major projects||2.38 (1.35)||2.17 (1.28)||3.75***||0.15|
|Led to publications||1.74 (1.18)||1.69 (1.12)||0.97|
|Gave skills to make transition from faculty to administration||1.95 (1.30)||1.81 (1.22)||2.26*||0.11|
|t Statistic||Effect Size
|Learning and Improving Practice Scale||3.32 (1.05)||3.39 (1.04)||1.11|
|Leadership/Change Scale||2.62 (1.13)||2.72 (1.17)||1.31|
|Networking||2.51 (1.09)||2.73 (1.15)||3.19**||0.14|
|Departmental Change||2.38 (1.08)||2.44 (1.14)||0.90|
|Institutional Change||2.04 (0.95)||2.13 (1.05)||1.32|
|Recharged and/or energized in work||3.26 (1.25)||3.29 (1.24)||0.31|
|Allowed to have fun in professional environment||3.11 (1.28)||3.07 (1.30)||0.54|
|Lent credibility for approach to teaching||3.00 (1.33)||3.13 (1.33)||1.68+||0.07|
|Lent credibility for approach to professional work||2.82 (1.34)||3.00 (1.33)||2.32*||0.10|
|Afforded opportunities to collaborate on projects||2.57 (1.34)||2.71 (1.36)||1.78+||0.08|
|Assisted in career advancement||2.29 (1.29)||2.40 (1.35)||1.48|
|Gave opportunity to pursue new grants or major projects||2.24 (1.32)||2.51 (1.36)||3.36**||0.15|
|Led to publications||1.71 (1.15)||1.80 (1.18)||1.24|
|Afforded opportunities to collaborate on projects||2.57 (1.34)||2.71 (1.36)||1.78+||0.08|
|Gave skills to make transition from faculty to administration||1.85 (1.25)||2.07 (1.29)||2.48*||0.12|
|NOTE:+p < .10; *p < .05; **p < .01; ***p < .001; aFive-point scale with 1=Not at all, 3=To some extent, and 5=To a great extent; bFOC=Faculty of Color|
These pronounced benefits for women and faculty of color suggest the importance of communities of transformation for these traditionally marginalized populations in STEM fields. With increasing efforts to diversify STEM fields and disciplines, these communities seem to provide increased support for these groups of STEM faculty than may exist on their home campuses. This is certainly reinforced by our interviews and participant observations throughout the project. We observed and were told repeatedly of faculty members feeling supported by both leaders and fellow participants in the communities, which frequently served as their main professional communities outside of their institutions. Often faculty members expressed much more colleagueship with people in the community than at their own institutions, which helped them remain in faculty positions despite the fact that they may have otherwise left academe without this support. For faculty of color in particular, the key benefits mentioned above point not only to the connections that can be made but also to the benefits that can accrue from these connections through involvement in these communities. Faculty of color gain significant social capital that provides access to grant and research opportunities, presentations that provide visibility, and networking/personal connections for letters and promotion.
Team Involvement for Organizational Change and Leadership Development
We also identified how the involvement of several individuals from a single institution increased departmental and institutional benefits, as well as boosting reported individual benefits for developing skills for leadership and change (See Table 6.3). In fact, one of the largest effect sizes we observed in our analyses in predicting departmental and institutional benefits came from having more peers from a single institution also involved in the community. Some of the communities of transformation in our study intentionally harness this effect through a campus team structure, in which campuses are encouraged to send teams of faculty and administrators to events to engage in group learning and reform. Yet, even in the communities that are not based on this team structure, we still observed benefits of involvement of multiple faculty members from the same institution.
By encouraging multiple faculty members from the same institution to pursue engagement in the same community of transformation, administrators can foster a common language and strategy of reform among their faculty members, which our findings suggest contribute to greater departmental and institutional change. These team members can learn together from other members of the community and gain ideas for strategies to apply to their home campuses. Additionally, faculty reported greater individual benefits in leadership and change skills when more of their peers were involved in the community, suggesting that being able to engage with one’s colleagues in these reform communities increases the ability to lead on their home campuses.
Additional Strategies for Broader Impact/Outcomes
While our study focused on the effects on individuals and how individuals influenced their departments and institutions, our qualitative data demonstrated further impact that we were not able to capture on the survey. In this section, we describe some of these important strategies/impacts that can result from supporting large scale CoPs. Section 9, in which we elaborate on the maturing phase of the CoP lifecycle and the focus on community expansion, also demonstrates some of the broader impacts that can be achieved by reshaping disciplinary groups, influencing larger institutional environments and whole sectors, targeting regional areas through regional networks, and even increasing international efforts through work and partnerships abroad. In this section we continue to describe the significant impacts of these CoPs that are often overlooked.
Creating other Networks or Communities of Practice
Some of the communities in this study have fostered the development or creation of other networks or CoPs for improving STEM education. PKAL in particular was responsible for establishing many other networks and communities of practice such as the Faculty for Undergraduate Neuroscience (FUN), the National Numeracy Network (NNN), and Quantitative Inquiry, Reasoning, and Knowledge (QuIRK). SENCER similarly works to create a network of informal educators focused on making science education more relevant.
Working across and between Communities of Transformation
Over time, the four communities of transformation have each co-hosted events with other STEM reform groups in an effort to broaden their impact by having more individuals attend. PKAL and BioQUEST have hosted several events together, including the BioQUEST summer workshop that we attended, which was hosted in conjunction with a PKAL regional meeting.
National Reports and Activities
These CoTs were also very involved with the creation of national reports and activities aimed at STEM reform. BioQUEST’s involvement with BIO 2010 (National Research Council, 2003) is an example of working with national organizations on a report aimed at creating large-scale pedagogical and curricular changes. Leaders in BioQUEST served on the planning groups, and they were instrumental in providing ideas stemming from their own philosophy and curricular materials. BioQUEST was also instrumental in revising the MCAT.
A Cadre of Leaders
PKAL’s Faculty for the 21st Century (F21) program was directly aimed at creating a set of leaders among STEM faculty that would advance into department chair, dean, and other leadership roles on campuses that would help to propel needed reforms over time. Recognizing that many STEM faculty neither aspire nor are prepared to play leadership roles, the program helped inspire and train these individuals. They were able to explore their passions in order to identify areas in which they might want to take leadership. These faculty were also provided training on how to manage departmental politics, how to create a shared vision, how to overcome challenges to creating change, and other skills important to implementing these important STEM reforms.
Service as Conveners across Groups
Particularly in the early years, there was little communication among different STEM reform efforts; the CoTs in our study played a pivotal role in convening related efforts and bringing together intermediary groups that could synergize their shared work. On numerous occasions, PKAL served as a convener by hosting science education groups across disciplines and associations.
Each of these CoTs offered different types of consultancies over the course of their histories to support other institutions or groups in STEM reform efforts. PKAL offered consultancies, supported by the Keck Foundation, that helped institutions pursue particular STEM reform projects, ranging from facilities reform to faculty and leadership development. SENCER offered “house calls” where SENCER leaders went out to campuses to help with curricular reform efforts and to coalesce faculty to work across different departments in support of new curricula. The POGIL Project provided consultancies for campuses that were interested in integrating the POGIL Project activities across a department or a set of departments.
Our study did not examine the relationship between certain design features and these broader educational impacts, but these CoTs seem to offer compelling examples for motivating this kind of change. PKAL, for example, appeared to be involved in activities that related to some of these broader impacts to their work on leadership development, to the creation of networks, and to the importance of convening. These areas all suggest directions that other future CoTs might want to explore if they seek to have these broader, enterprise impacts.
This section describes the outcomes and benefits of participation in STEM reform CoTs, while also highlighting additional strategies for achieving broader outcomes for STEM reform. Our findings reveal that these communities contribute most frequently to individual outcomes related to feeling reenergized in one’s work, learning and improving practice, and gaining credibility for one’s work, yet they also contribute to broader outcomes for faculty members’ departments and institutions in terms of broader curricular and pedagogical uptake. These communities have also contributed to broader change through other strategies we highlight, although it is hard to capture these impacts empirically. We turn in section 6 to the ways in which these communities are designed to contribute to these outcomes and benefits.
7. Tables 5.1 and 5.2 list the factor loadings and Cronbach α values for each scale. The factor loadings for each item represent the extent to which each item is correlated with the underlying factor, with a value of 1 being the strongest possible relationship to the factor. For example, the item most strongly correlated with our first individual outcome Learning and Improving Practice is the item “Led to changes in teaching practice.” The Cronbach α value represents the internal reliability of the items within the scales, again with 1 representing the strongest internal consistency of each scale.