Abstract
The use of online instruction for undergraduate STEM courses is growing rapidly. While researchers and practitioners have access to validated instruments for studying the practice of teaching in face-to-face classrooms, analogous tools do not yet exist for online instruction. These tools are needed for quality design and control purposes. To meet this need, this project developed an observational protocol that can be used to collect non-evaluative data for the description, study, and improvement of online, undergraduate STEM courses. The development of this instrument used a sequential exploratory mixed methods approach to the research, design, pilot-testing, refinement and implementation of the protocol. Pairs of researchers tested the final version of this instrument, observing completed online undergraduate STEM courses. Across 2,394 pairs of observations, the observers recorded the same indication (yes or no to the presence of some course element) 1,853 times for an agreement rate of 77.4%, falling above the 75% threshold for an acceptable level of agreement. There was a wide range in the inter-rater reliability rates among items and further revisions were made to the instrument. This foundational work-in-progress instrument should be further developed and used by practitioners who are interested in learning about and reflecting on their online teaching practice.