Talk:Developing Effective Multiple Choice Items

From KNILT

Mod 11: Peer Eval -- Bwest (talk) 13:22, 21 November 2014 (EST) -- Bwest (talk) 13:22, 21 November 2014 (EST)

Comments on Unit 2: Item-Writing Flaws - Brandon West - November 21, 2014

Evaluation Criteria
Learning Objectives

Your unit objectives align with the first two lessons of unit 2. It would be helpful for you to mention the performance objectives on the page titled "Item-Writing Flaws: Lesson 2" page, as I couldn't easily identify what participants were supposed to be doing on each unit without consulting your portfolio page.

Content-Goal Consistency
Your content matches your unit objectives. Listing the performance objectives on the Unit page (or on each lesson) would help demonstrate consistency between content and goals.

Instructional Sequencing
The sequencing of the unit makes sense. I would need to see the completed unit to give more feedback, however.

Engagement and Interaction
I like how you used self-reflection at the beginning and end of lesson 2. Asking participants to draw on the content from lesson 1 was a smart move. I would like to seem some type of engaging/interactive element (self-reflection or otherwise) integrated into lesson 1 that explicitly tells the participant what to do.

Two of your performance objectives mention correcting flawed test items (5. Given a flawed multiple choice item, participants will be able to identify the type of error in the item with 80% accuracy. 6. Given a flawed multiple choice item, participants will be able to rewrite a correct version of the item with no flaws). These might make for interactive activities at the end of Unit 2.

Technical Quality
I do not have any technical issues to report. I appreciated having the navigation at the bottom of the lessons. I think that incorporating some graphics would spruce up the homepage. Consider using images licensed under the Creative Commons. I think you should include a description in the frames of your images. This would be helpful for the participant in lesson 2, where you compare and contrast different test questions.

A small comment about your mini-course homepage: You provide a link to the individual lessons under each unit. I think this may discourage participants from clicking on the Item-Writing Flaws: Lesson 2 page, which serves as an introductory page to the unit. Consider forcing participants to land on this page, so they read the context of the unit and can read the unit and performance objectives, if you add them here.

Extended Resources

I assume that their will be extended resources in the lesson 4, but I could not evaluate this component at the time of review.
Other Comments

As a librarian, I have to ask where you got your test questions and other content from, so consider possibly including a references section at the bottom of each lesson. If you use images, you will want to cite those as well. What I have done is cite everything I will be using in my mini-course on my portfolio page. I include the references for content at the bottom of each unit, but only cite the images on my portfolio page.

Overall, this mini-course is off to a great start - it's definitely one I am interesting in reviewing once it's done!



Ray's Response - Nov 21

  • Will include: objectives listed on unit pages, alt text for images, CC licensed graphics, forced navigation to the Unit 2 page*, links to extended resources throughout lessons (mostly at the end of each unit like you mentioned), reference sections for each individual lesson.
  • With Unit 2: Lesson 1, my goal is really to familiarize participants with the terminology for the subsequent three lessons. I don't want to spend an inordinate amount of time on simple terms and definitions (my written objective doesn't really capture that). Were you thinking you wanted more in order to fulfill the objective I have, or do you think that participants would need more time / thought to solidify that vocab?
  • The self assessment at the end of Lesson 2 was meant as a first look at the objectives for identifying flaws and re-writing items. I'll include a more comprehensive version of that at the end of the unit that draws from conceptual material in Lessons 2-4.
  • Citations: I wrote all sample questions and took screenshots from MS Word. Do you happen to know if/how I should cite that? Also, most of the conceptual material so far is based on the Linn and Miller text (cited on portfolio page). I'll put that in reference sections by lesson also.

Thanks for the good ideas and thoughtful analysis. I appreciate it.

instructor comment on your mini-course -- Jianwei Zhang (talk) 15:32, 1 December 2014 (EST)

This mini-course is well structured and the structure of the units maps consistently onto the learning objectives identified. The sample unit on common flaws is very solid and engaging, with good examples and self-reflection/assessment.

One issue: Item Writing Flaws: Lesson 1 is about the structure of a multiple choice item. It should probably go to your first unit, near the beginning of this course. What do you think?

Please continue your good work to finish the rest of the units.

Anatomy of a MC Item First -- Ray Boss (talk) 00:22, 2 December 2014 (EST)

That sounds like a good idea. That lesson is really just establishing basic teminology, and it will be useful in Unit 1 also. Thanks for the suggestion and the comments. - Ray