Skip to main content

Data Analysis Workshop - Item Analysis

Data Analysis Workshop - Item Analysis

Overall learning goal

The goal of the data analysis workshops is multi-layered. By carrying out a more in depth analysis of exam results, CARE members become more familiar with the Using Data Process components and tasks, and are introduced to the Data Coach’s Guide’s data pyramid, in order to drill down into five detailed levels of data analysis. They also gain familiarity with the Collaborative Inquiry process. To highlight the findings of the day’s analysis as well as recommendations for moving forward, the task based workshop’s final goal is to produce an annual provincial report.

Synopsis

Following a Data Analysis Workshop to examine the five uniform provincial ministerial exam results, the CARE members and the EBP Project gather annually for a second Data Analysis Workshop, digging deeper into individual exam results.  

The purpose of the Item Analysis workshop is to support the CARE members in a deeper analysis of individual exam items and student samples. Following a revision of the Using Data Process components and tasks, participants are introduced to the data pyramid as detailed in the Data Coach’s Guide, in order to drill down into five detailed levels of data analysis. They also explore the roles, norms and protocols of the Collaborative Inquiry process.

The task based workshop ends with an annual provincial report written by the group to highlight the findings of the day’s analysis as well as recommendations for moving forward. The report includes a table of results for all school boards.

Instructional design

Introduction

The EBP facilitator introduces the day’s proceedings and connects the current task to the previous workshop on using the data analysis process to create a provincial portrait of the 5 uniform ministerial exams. 
The Item Analysis workshop is broken down into two parts; Item analysis, where participants form small subject groups to analyse results of specific exam items, and a student sample exercise where teams sort, code and compare results from an exam.
The role of Collaborative Inquiry (CI) in the Using Data Process is also introduced. Members will become familiar with the CI process for Data Analysis throughout the workshop with a view to supporting school teams with the process.

Development
PART 1: DATA ANALYSIS

The morning session involves CARE members working through a four step process to analyse individual test items in order to identify specific strengths and weaknesses in student performance.

Step 1: Prediction

Step 2: Inventory of pass/fail by items

Step 3: Data analysis: Go visual + Observation

Step 4: Write up for provincial report

Using the DBMS database, participants highlight pass/fail, count the failed items, and identify differences before zooming in to look at proficiency for each item. Following the item analysis, participants connect to the nature of exam items and use the data driven dialogue process to analyse the results in more depth and make inferences.

 

To conclude the morning session and the four step process, Participants draw together the day’s findings to draft a description of the results of the item analysis, including a summary of the findings, the most successful items, the weakest strands, and other relevant data.

PART 2: STUDENT SAMPLES

Using the outcomes of the data analysis again, CARE members start the afternoon by selecting a student sample and creating an inventory by connecting to the item analysis results. Participants identify weak questions and detail what the students are able or unable to do by coding and categorising types of mistakes, examining missing answers for patterns and using an inquiry mindset throughout.

Time permitting, and using the morning’s item analysis results and the student samples, participants draft student learning problems that would be worth exploring further.

The next step is to review the possible causes of student learning problems (as detailed in Verifying Causes Tree diagram), with a view to identify the causes of each before putting in measures to rectify the problem and improve student results.

PART 3: WRITE UP & COMMUNICATION

Members begin to write a descriptive report of the results from the above analysis in order to begin drafting the provincial portrait of how students are doing in the ministerial exams overall and across each school board.

The arising report is distributed to DEEN, subject sub-committees and to each of the represented school boards.

Conclusions

CARE members feedback a summary of what is included in the report. The key ideas from the day’s learning and tasks are reviewed and the group look ahead to the next steps.

Evaluation

At the end of each workshop, members are asked to complete and hand in either exit cards or evaluation sheets, commenting on what they would like to address at the next session.