Abstract
SETTING: A public health society working group wanted to use social marketing approaches to engage with a community and stimulate social support for a health treatment. The group struggled to collect effectiveness (summative) data during the project. To make up for this lack, the group explored ways to measure effectiveness of engagement (the primary outcome) based on written records (meeting minutes) kept during the project. INTERVENTION: The working group kept minutes of meetings that contained records of the level of participation of members by names. The text of 18 meetings (14,000 words) was edited so that the names of participants were replaced with roles that corresponded to working group members' roles: grassroots health advocates, community health agency representatives, and experts or knowledge leaders. The corpus was imported into a text analysis platform that measured word frequency. Results were tallied for the three categories of group member roles. To validate the method as a meaningful summative evaluation, the text analysis approach was critiqued using a developmental evaluation framework. OUTCOMES: The text analysis evaluation indicated that the word frequency of "partner," (community health partner), "community" (grassroots health advocates), and "expert" (or knowledge leaders) tags began to converge as the campaign progressed. Initially, experts and community health partners spoke less in meetings, and community members spoke more. Over time, all members began contributing more equally during the meetings. The checklist evaluation indicated alignment of the technique with established evaluation protocols used in the field of public health. IMPLICATIONS: The text and checklist analyses support the notion that engagement among working group members may be, and thus may be seen as, a precondition of engagement with the community. When used with evidence from event evaluations, the innovation may be used as an argument for effectiveness as an outcome in community-based public health campaigns that do not use conventional project (summative) evaluations.