Observation Validity Working Group Meeting

About 

This gathering of mathematics and STEM education researchers was designed to bring together experts in classroom observation protocol development for use in school settings to evaluate fidelity of implementation of interventions/programs and to gather formative input to improve practices.

 

Over the course of the two-day meeting, the group shared projects and discussed the following topics:

 

·  highlights and challenges of measurements in individual projects, identifying similarities and differences,

·  features of measures and measurement development and pros and cons of their usage across users and time,

·  validity evidence and the essential claims we put forward about it,

·  how measures support instructional improvement and what that means for researchers developing and collecting data on the constructs being measured.

 

Questions based on attendees’ experiences and projects.

 

·      Who is our audience? Researchers? Administrators?

·      Does validity in measurement matter if reliability is in question?

·      How do you identify good instructional decisions when a single tool doesn’t capture everything?

·      How does a team define rater agreement? Rater consistency?

·      How do you differentiate rater agreement vs. rater reliability? What about rater drift?

·      What opportunities exist to improve rater reliability?

·      How does a team operationalize the construct being measured?

·      Is the observation tool the intervention? Does the tool measure instructional quality?

·      Does the observation tool become too proprietary by the developer to be used effectively?

 

Considerations around the uses of measures emerged and prompted the following discussion points:

·      collecting and analyzing observational data (including multiple time points),

·      using existing measures and development/how-to-guide or approaches to avoid re-inventing the wheel,

·      using measures across users,

·      using measures to provide feedback to teachers,

·      interpreting dis-confirming evidence,

·      determining relative importance of various sources of data,

·      embedding technology within a measure, and

·      ensuring relevant and appropriate use of measures.

Date

March 2-3, 2020

 

Hosts:

  • Dr. Leanne Ketterlin Geller, Professor and Texas Instruments Chair in Education; Director, Research in Mathematics Education, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Elizabeth Adams, STEM Evaluation Researcher at Research in Mathematics Education, Simmons School of Education and Human Development, Southern Methodist University

About our Experts

  • Dr. Courtney Bell, Director, ETS/WCR at Wisconsin-Madison
  • Dr. Julie Cohen, Assistant Professor of Curriculum, Instruction, and Special Education, University of Virginia
  • Carlin Conner, Research Assistant, Education Consultant, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Jillian Conry, Research Coordinator and Instructor, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Angela Crawford, Postdoctoral Researcher, Research Project Coordinator, Boise State
  • Dr. Richard Duschl, Executive Director, Caruth Institute for Engineering Education, Southern Methodist University
  • Dr. Evelyn Johnson, Scientific Director, Professor, Department of Special Education, CEO at Lee Pesky Learning Center, Boise State University
  • Dr. Nathan Jones, Associate Professor, Program Director for RS Courses, Boston University
  • Erica Mason, Graduate Research Assistant, Department of Special Education, University of Missouri
  • Karen Pierce, Research Assistant, Research in Mathematics Education, Simmons School of Education and Human Development, Southern Methodist University
  • Robyn Pinilla, Research Assistant, Research in Mathematics Education, Simmons School of Education and Human Development, Southern Methodist University
  • Tiffini Pruitt-Britton, Graduate Research Assistant, Research in Mathematics Education Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Anthony Sparks, Research Coordinator, Research in Mathematics Education, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Paige Ware, Associate Dean and Professor, Department of Teaching and Learning, Simmons School of Education and Human Development, Southern Methodist University
  • Ann Marie Wernick, Graduate Student, Southern Methodist University
  • Dr. Jeanna Wieselmann, Research Assistant Professor, Caruth Institute for Engineering Education, Southern Methodist University
  • Dr. Annie Wilhelm, Associate Professor of Mathematics Education, Department of Teaching and Learning, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Jonee Wilson, Assistant Professor, Mathematics Education, NC State
  • Sumei Wu, Research Assistant, Simmons School of Education and Human Development, Southern Methodist University
  • Dr. Paul Yovanoff, Professor, Department of Teaching & Learning, Simmons School of Education and Human Development, Southern Methodist University