Planning a SoTL lesson study: Step 2 - Data collection and analysis

Identify student learning questions

In our project, we collected data before, during and after a lesson to discover students’ knowledge, correct and incorrect thinking, skills and motivation. You can also gather student feedback on how the lesson was delivered and their experience.

Some types of lesson study questions that you can ask include:

• Does instruction help students learn an important concept or skill? What student learning gains are greatest?
• How do students’ approach learning a concept or skill? How does this compare to how experts demonstrate knowledge or skills?
• Where do students struggle when learning? What alternative ideas or misconceptions stand in the way of student learning?
• Are students motivated to learn certain concepts or skills? What activities or types of instruction increase or decrease student motivation?

Plan data collection and analysis for before, during and after the lesson

Once we determined a set of questions to use, we identified what types of assessment techniques would allow us to answer these questions and also gather feedback on the lesson (Table 1). For example, we collected both qualitative and quantitative data to gain understanding of how the lesson is impacted students learning and understanding of the topic.

One of the most important things when planning data collection and analysis is to make sure that you consider how much time is needed to both collect and analyze. Quantitative data can be faster to collect and analyze, but can also be difficult to understand without context. Qualitative data can be more time-consuming to analyze, but can add important information and understanding on student thinking. There are many resources available on data analysis. The Taylor Institute for Teaching and Learning also holds workshops that focus on data analysis.

Table 1. Examples of data that can be collected before, during, and after lesson to explore student learning.

Data that can be analyzed quantitatively

• Multiple choice questions
• Clicker questions
• Likert questions
• Some observation protocols (e.g., COPUS Classroom Observation Protocol for Undergraduate STEM)

Data that can be analyzed qualitatively

• Written questions (worksheets, surveys, assignments)
• Essays, research papers, and projects
• Portfolios or laboratory notebooks
• Presentations
• Minute papers
• Written reflections
• Think aloud interviews, focus groups, student meetings or discussions
• Classroom observation

Recommendations for data collection

• Your study must have REB-approval before collecting data that you will share; you must obtain student consent before gathering any data.
• Check that the plan to collect data is well aligned with the lesson plan learning objectives and will provide a useful set of data for analysis and interpretation.
• Before generating new assessment materials, search the literature to find available concept inventories and motivational surveys that have evidence of validity and reliability. Generating surveys and collecting evidence of validity and reliability is a separate multi-year project!
• If you are giving the lesson, have someone else collect the data.
• Plan to take detailed notes on how the lesson was carried out and write down all ideas that arise during the lesson.
• Deidentify all data following collection and before analysis.

Reminders

• Ensure you have all the tools you need, including additional help to record observations, before collecting data during the lesson.
• Take notes on what worked, what didn’t and what could be done better in the future as you collect data.

Recommendations for data analysis

• Deidentify all data before analysis. Remove any data from students who did not provide their consent.
• You can search the literature to find published lesson studies that have analyzed the same kind of data for examples and comparisons.
• Check that the analysis methods align with the type of data collected (e.g., numerical, ordinal, categorical, etc.).

Reminders

• Take careful notes on how you carried out each analysis.
• Keep raw data separate from data you are actively manipulating and analyzing.
• Consult with experts for help with analyses.

Conclusion

In this blog, we have written a description of the “Data Collection and Analysis” stage of Three Steps to Planning a SoTL Lesson Study.

In the sections included in this blog you can find information about:

• Generating a research question
• Data collection
• Data analysis

Each step includes our recommendations to help save you time and effort and make sure that you will generate results that you can share.

We have also described Lesson Plan and Study Design and Sharing and Disseminating Work in separate blog posts.