Interview validation of the Physics Lab Inventory of Critical Thinking

K. N. Quinn, C. Wieman, and N. G. Holmes

Stanford University

https://arxiv.org/pdf/1802.02424.pdf

Recent work in physics undergraduate education examines the benefits of various teaching styles.  In order to do this, there must be a measurable effect on student outcomes.  Performing well on a course test is a familiar metric, often equated with student understanding.  However, course tests often must be created quickly and tend to test recognition and recall.  Although much more time-consuming, it is possible to create a concept inventory, a standardized test that determines specific incorrect lines of reasoning and is statistically validated.  The Force Concept Inventory (FCI) is a famous example in physics (Hestenes, Wells, and Swackhamer, 1992).

Quinn et al. are developing a measurement tool called the Physics Lab Inventory of Critical thinking (PLIC) to assess students’ critical thinking.  The authors define critical thinking as the ability to identify whether conclusions are supported by evidence, and to distinguish significant effects from random noise (Holmes, Wieman, and Bonn, 2015).  They note that although critical thinking is not physics-specific, it is often context-dependent and so should be tested in a discipline-specific way.

Their paper gives an overview of the creation process for the concept inventory.  First, the scope and topics of the assessment are established.  Next, open-response questions on those topics are created and answered by students.  These answers are used to determine probable misconceptions about the topic, and the corresponding student responses.  This allows the creation of relevant distractors for multiple-choice questions.

The test is then revised and calibrated.  Students are interviewed to uncover the reasoning behind their responses.  Answers may be added, deleted, or reworded to make them more discriminatory.  Next, questions are calibrated for difficulty by comparing to true experts, to see what they would choose.  These interviews, revisions, and retesting may be iterated to improve test sensitivity and reliability.

In the case of the PLIC, the authors test students’ critical thinking by asking for possible actions in response to a poor lab experiment outcome.  Students followed three main behaviours.  One was to do as much as they could, relatively aimlessly.  Another would be to select actions that were described using familiar keywords and phrases, whether they were relevant or not.  The third would be to weigh options critically and prioritize them.  Interestingly, as the response options became more restricted or presentation of the question became less familiar, the first two groups would shift into more discerning behaviour.

Once the authors create a scoring scheme based on the answers given by experts, they plan to create a final version of the PLIC for widespread distribution and use.