Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
Test items: if I can't tell the difference between 80 and
90% proficiency, how do I set criteria?
Author:
Kristi Holsinger
Posted Date:
October 30, 2012 4:04 PM
Hi Team,
Dave has mentioned multiple times
that on our objectives, if we can't determine the difference between 80% and
90%, why are we using that criteria? I absolutely understand his rationale, and
I can absolutely hear my trainees voicing their concerns with this kind of
evaluation. I just don't know how to fix it!
Here's an objective I could use some
help fixing:
Given an image of black, white, and
green electrical wires, learners will explain how power reaches a light fixture
with 80% accuracy.
Thanks for your help!
Kristi
Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
specific number of examples in criterion?
Author:
Rhonda Schalk
Posted Date:
October 30, 2012 9:41 PM
Here's
your objective: Given an image of black, white, and green electrical wires,
learners will explain how power reaches a light fixture with 80% accuracy.
In trying to explain a fact objective in another post, I think I may have
come up with a helpful idea. (Your call on whether it truly is helpful or not!)
I used an assessment criterion of something like "with at least 3
examples". So, thinking of your instruction, you might give your learners
5 points of how the power reaches the light fixture. Then say that they need 4
out of the 5 points to say they've mastered the objective (thanks Micah for
pointing that out!). Since I don't know how the power reaches a light fixture,
I don't really know how many points they would need to make. It's magic! LOL
Does this make it more confusing????
Rhonda
Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
RE: specific number of examples in criterion?
Author:
Kristi Holsinger
Posted Date:
October 31, 2012 3:19 PM
Hi Rhonda,
That's great advice, thank you!!!
Kristi
Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
Author:
David Knowlton
Posted Date:
October 31, 2012 4:35 PM
I agree with Kristi that this
objective does need some work. I'm glad that Rhonda offered a potential
solution. How about something like this (keep in mind that I don't
understand the content).
Uh . . .
Yeah, I'm struggling to fake my way through
it because I don't know the content. I'm trying to say something like
this:
Given a blank piece of paper, the
learner will explain how power reaches a light fixture that includes a
consideration of the black, white, and green wires.
Does that help?
Maybe your "explanation"
doesn't really fit Morrison's model. Does he say that
"explanation" is a valid test? What does Morrison call that
type of test question? How does he say to assess it? So, I just
looked in the chapter on test items: essay tests versus short answer
tests, for instance. Would declaring which type of test you think this is
help you decide how to assess it?
Maybe you need to rethink the entire
objective to multiple choice or true and false something:
Given five statements about the role
of the black, white, and green wires in powering a light fixture, the learner
will identify three of them as being true.
Dave
Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
and labeling the test question...
Author:
Rhonda Schalk
Posted Date:
October 31, 2012 5:14 PM
If Kristi
went with Dave's suggestion for her objective of: Given five statements about
the role of the black, white, and green wires in powering a light fixture, the
learner will identify three of them as being true.
Then, would her test question then be labeled: Objective test/multiple choice
(or maybe true-false depending on how she actually worded the question/s)?
Rhonda
Thread:
Test items: if I can't tell the
difference between 80 and 90% proficiency, how do I set criteria?
Post:
RE: Rhonda has rocked my world this week!
Author:
Kristi Holsinger
Posted Date:
October 31, 2012 7:12 PM
Hi Rhonda,
Thanks for all of your great
ideas--truly can't thank you enough! This ties into one of my biggest projects
after listening to Dave's MP3: labeling test questions.
Kristi
Thread:
Fact recall or concept application?
Post:
Fact recall or concept application?
Author:
Kristi Holsinger
Posted Date:
October 30, 2012 4:07 PM
Hi Everybody,
In labeling an objective, I just
want to check and see if I have fixed this one correctly.
Objective: Given a diagram of an
electric box, the learner will explain how to safely deactivate power to the
home with 100% accuracy.
I had this listed as fact-recall,
but I think it's concept application. Am I right?
Thanks,
Kristi
Thread:
Fact recall or concept application?
Post:
Author:
Rhonda Schalk
Posted Date:
October 30, 2012 4:56 PM
Hey
Kristi,
I went back to the book to check out what it said. I think that your objective
would be procedure application. The book says, "A procedure is a sequence
of steps one follows to achieve a goal" (p. 126). So, isn't deactivating
the power a sequence of steps? First you go to the power box, then you flip the
switch, then you check to see if the outlet is getting power using the little
checking thingy. Right? (Of course you would use the correct terminology)
The book said, "concepts are categories we use for simplifying the
world." So I don't really think that this falls under that. Maybe if your
objective was "Given a picture, the learner will be able to identify the
circuit breaker box with 100% accuracy." then that would be concept-recall
because they are identifying the box. I can't think of one for concept
application right now, sorry.
I hope I'm not giving you false info and not confusing issues!
Rhonda
Thread:
Fact recall or concept application?
Post:
Author:
Micah Day
Posted Date:
October 30, 2012 4:59 PM
Hey Kristi,
It sounds to me that your original
label was correct. Within this objective, you are solely mentioning how
to deactivate an electric box, correct? If you were to have mentioned
more than one box, then I could see you changing it to concept.
Also, explaining how to deactivate
the power, this sounds more so of a simple recall. However, if there are
multiple steps involved within this explanation, then I would say application
may be a better fit within this objective classification.
Micah
Thread:
Fact recall or concept application?
Post:
Author:
Rhonda Schalk
Posted Date:
October 30, 2012 9:29 PM
I'm pretty
sure that Amber's objective isn't fact-recall. (You may have posted before you
read my other reply, but I'm going to try to make my point again in a different
manner anyway. :-)
Here's her objective: Given a diagram of an electric box, the learner will
explain how to safely deactivate power to the home with 100% accuracy.
The book says, "A fact is a statement that associates one item with
another" (p. 125). So if her objective were something like "Given a
piece of paper and pencil, the learner will explain why the electricity needs
to be deactivated prior to the repair naming at least 3 reasons." then it
would be fact-recall, I think. I don't think this would be a concept because the
book says that concepts are categories. My objective wouldn't be principles or
rules, either because they are "relationships between concepts".
Couldn't be procedure because those are steps. It's pretty obviously not
interpersonal skills or attitude.
So, like I said in a previous post, I really think that her objective is
procedure because it says "how to". I also think it's application
because they are using the diagram to help explain it. Although I've been
questioning myself about that part.
Rhonda
Thread:
Test
items: include a description of the type of assessment question
Post:
Test items:
include a description of the type of assessment question
Author:
Kristi Holsinger
Posted
Date:
October 29, 2012
7:18 PM
Hi
Everybody,
This
is a follow-up question from last week. Dave asked "WHY" are we
adding certain elements to our project, and there is one element that is still
a mystery to me:
Do you understand WHY the
rubric asks you for the test items to include a "description of the type
of assessment question"? Do you understand how to do that?
I'm not clear on how to
answer either of these questions.
Thanks for the help!
Kristi
Thread:
Test
items: include a description of the type of assessment question
Post:
Author:
Rhonda Schalk
Posted
Date:
October 30, 2012
7:23 PM
I
briefly went back to the examples on reserve and didn't really find answers in
these. But I did find that chapter 12 answers these questions, kind of.
Kristi asked about Dave's questions: Do you understand WHY the rubric asks you for the test
items to include a "description of the type of assessment question"?
Do you understand how to do that?
I think the reason that Dave is wanting us to include the "description of
the type of assessment question" is to make sure that we are really
thinking about why we are using this type of question for our assessment. The
book says this, "A direct relationship between instructional objectives
and test items must exist. Thus, it is customary to derive test items directly
from the objectives" (p. 302). So the verb of the objectives gives us the
clue as to what type of assessment we need to apply.
Here is an objective from Kristi's project:
Given an image of black and white electrical wires, learners
will explain how power reaches a light fixture with 100% accuracy.
Task Analysis: VI:D—E.3
Principle: Concept application
The
verb in this is "will explain". by looking at the examples on p. 302,
it says, "To state or describe: writing or speaking a short or lengthy
answer." For her labeling of her assessment, then she should be
"Constructed-response" and her test question should be one of the
following: short answer, essay, or problem solving.
Does
this help, Kristi?
Rhonda
PS I now need to go back and make sure that my
questions match in this manner! Ack, more work!!! :-)
Thread:
Test
items: include a description of the type of assessment question
Post:
Author:
Jami Trusty
Posted
Date:
October 30, 2012
9:11 PM
Edited
Date:
October 30, 2012
9:14 PM
Status:
Thanks Rhonda, this really
helped me. I was a little lost as to "the description of the
assessment" myself. Your explanation was very helpful - good
reference back to the text. I guess I will add this to my never-ending
list!
Thanks.
Jami
Thread:
Test
items: include a description of the type of assessment question
Post:
"Verb
Implications" PP was at my side
Author:
Scott Esker
Posted
Date:
October 31, 2012
8:17 AM
Status:
Rhonda,
Great reply! In addition to
what the book says about labeling and designing an assessment, Dave has some
great PP's that helped me out tremedously as I was desiging my documentation.
There is a PP called
"Verb Implications" that guides a designer in choosing an assessment
based on the verb that is used in the objective. The PP lists some common verbs
in the objective then goes on to offers suggestions for the assessment type. In
other words, it sort of offers a prescription of assessment based on the action
verb used.
The "Verb
Implications" PP can be found here (if the link doesn't work, try
following the learning object for test items under week 5):
This had this PP at my side
(literally, I had it open on my ipad) as I designed the assessments for my
documentation.
Scott
Thread:
Test
items: include a description of the type of assessment question
Post:
RE: matching
test to objective
Author:
Kristi Holsinger
Posted
Date:
October 31, 2012
3:20 PM
Hi Rhonda,
You're awesome! Thanks so
much!
Kristi