Instructional Design Rubric

In looking around for research opportunities, I made an indirect connection between the research a friend of mine was doing and something I had been thinking about at work. I helped with the research on a pilot study. It was using a rubric (Boote & Beile) to measure the quality of dissertation literature reviews. It turns out literature reviews aren't taught consistently, and (no surprise here) are not consistently performed properly either. I'll just say that after reading a stack of dissertations, my confidence in my own ability to do academic research increased dramatically.

The conversation has been had several times about how strange it was for my friend doing a literature review on doing literature reviews, just as last summer I was working on designing instruction about designing instruction. My thoughts turned to whether it would be possible to do something at a level that could improve more of the higher education world, of which PhD studies are a fairly small part. One interesting idea I came up with deals with an analysis of the gap between expected outcomes from current students in higher education and recognized outcomes from graduates of higher education, with cross sections based on type of institution, level of degree, subject area, etc. My guess is that we'd find that students aren't fully realizing the benefits they expect to receive; that is, there is a misalignment somewhere. Think about what you were asked to do in your first job out of college and whether you used anything that was listed as an objective for any of the courses you took. While I still think that is an interesting topic to pursue at some future date, it's not quite dissertation material at this point.

At work, we follow a fairly well structured design process to come up with what we call the competencies and learning objectives for our courses. Competencies are roughly equivalent to educational objectives in Bloom's Taxonomy, and the more focused learning objectives match up with Bloom's instructional objectives. Going with a more wide angle, we also couch everything for a degree program within a conceptual framework, which Bloom would probably call global objectives.

While there is research on Bloom's Taxonomy, much of what I've seen has to do with validating the order, leading to a recent swap in the last two levels since researchers had identified that evaluation is less cognitively demanding than creation. There are other taxonomies out there, which overlap quite a bit, but I have yet to come across any good research that really supports the most famous of educational taxonomies. I believe it, because it makes sense, but who is to say if something is missing or if all the current components are really necessary? One article talked about how the iconic triangle should be flipped on its point so it opens up as you reach higher cognitive levels. Whatever. I don't care about the job aid. I want to know if the taxonomy is right. So the proposed research is to rate the quality of a course designed using Bloom's Taxonomy and compare that to some measure of the actual quality of the finished course developed according to that design.

I haven't pulled this together as a formal literature review yet (but I will be pulling out my Boote & Beile rubric when I do). Just a quick compilation of features from several models out there, and I come up with an unpolished rubric that I hope may be useful in rating the quality of a course design, with design being the first D in ADDIE, not the kind where you're choosing background colors.

While I have not yet found anything substantive related to measuring the quality of the initial design of a course, Merrill does have his 5 Star and e³ rating forms based on his First Principles of Instruction along with some other theories, which are intended to rate a developed course. At that point, however, it's a little late. The course is already created and perhaps already taught. It would be more helpful to rate the design of a course before it's developed. Fix the problems before developing the instruction on a flawed design.

Again, this is not complete or valid in any way. The models I considered, based on our practice at work and general research I'm familiar with in the area, in no particular order, are as follows:

• Wiggins & McTighe - Backwards Design Model
  
• Merrill - First Principles of Instruction
  
• Gagne - Nine Events of Instruction
  
• Anderson & Krathwohl - Revised Bloom's Taxonomy
  
• Mager - Preparing Instructional Objectives
  
• Knowles - Principles of Andragogy

The principles or characteristics of effective design that align with the above theories, and that serve as a first draft of rubric principles are:

• Problem-based or realistic (publishing work, problem solving, creation)
  
• Alignment between assessment and content (teach what you test and test what you teach)
  
• Appropriate path to complete objectives (flexibility for diverse learners, no orphans, properly scaffolded)
  
• Measurable behavior (specific, discrete, observable)
  
• Level of performance (alignment with content and audience)
  
• Clarity (one verb, understandable, meaningful, maybe combine with measurable)
  
• Metacognition
  
• Bridging prior knowledge and future practice
  
• Any other suggestions as to what's missing?

I'm picturing a four point scale, 0-3 for each item. The 0 is obvious; it simply doesn't meet the requirements. A score of 1 would indicate minimal coverage. A score of 2 would indicate significant alignment but missing higher level components or perhaps most objectives are written correctly and a few incorrectly. The goal is to score a 3, meaning all principles and characteristics are present and appropriately applied.

The rubric itself would take the form of a table, which I'll put together later, but the principles or characteristics would be listed down the left and the points across the top, with a description of what is needed to receive that score in the intersecting boxes.

A graduate-level course, for example, with all multiple choice tests would likely score poorly on problem solving and realism but could score okay on the alignment if you're planning on teaching at the scantron level. One could argue whether completing a multiple choice test is measurable behavior; I'd maybe put that in the 1-2 range, since while the correctness of the student's answers can be easily measured, it's debatable whether it can be considered behavior. The level of performance is obviously wrong if we're talking about giving grad students multiple choice tests, so we're talking maybe a 1 here if the questions get past a superficial level. There's not going to be much metacognition going on.

For an undergraduate-level course, on the other hand, while some of the principles may still be rated low in terms of realism and metacognition, the level of performance may be appropriate. It may even be possible to score decently on bridging prior knowledge and future practice if the scantron content is considered foundational knowledge that will support more in-depth performance in later classes.

Good, bad, too much, not enough? Other principles, theories, or methods that should be included? Has anyone actually done something like this for instructional design? What measures other than inter-rater reliability could be used to validate the score from this type of rubric?

Metered Internet

In the early days of the Internet, some 10-15 years ago, there was a different landscape than we view before us today. Consumers and content providers faced a strange problem. Many consumers used slow but cheap dial-up connections to access the Internet. They couldn't justify the outrageous costs of high speed Internet, which probably wasn't even available where they lived. Maybe if there was actually any good content beyond plain text and some banded gifs, it would be worth paying the higher price, and I'm not just talking about those Flash splash pages that were popular back in 1999. I mean real multimedia content.

Content producers didn't want to invest in putting heavy content online, however, because nobody could see it. It would take too long to download over a 56K connection; there really wasn't an audience with the necessary bandwidth, other than perhaps faculty and students at universities.

We talked about it, though. Some day in the future, we would all have broadband access, and we would be able to watch videos whenever we wanted. We would make video calls, share music, and play games. As the content grew little by little, so would the numbers of broadband subscribers until there was a critical mass.

High speed Internet is practically considered a public good now, and its pervasiveness is taken for granted. Interactive multimedia content abounds, broadband access is available in most homes, as well as on mobile devices, and many consumers produce large amounts of their own content.

So of course we are now taking the obvious step of reducing our broadband use through throttling and overage charges.

In Canada, there have been some crazy expensive overage plans that look like they are being scaled back due to their ridiculosity. I can't find the link right now, but one blogger calculated that between speed limits and overage charges, it was both faster and cheaper to buy a new terabyte hard drive, fill it up, and overnight it across the country than it would be to transfer the data over the internet.

These data caps are advertised as necessary due to the strain from multimedia content, especially video sites like youtube, hulu, netflix, and the rest of their ilk. That sounds reasonable until you look at the way service providers are bundling internet-based phones and TV along with their internet service. So yes, that's right, the service providers are piling more types of traffic onto the very same network that they claim is not big enough for current usage.

The method of measuring by amount of data transferred over a month is flawed anyway. Data transfer should only really be measured by available speed. If you want to stream voice and video, you need a higher speed than if you just tweet and check emails with little real regard to volume.

Mobile is following the footsteps of traditional ISPs. Many mobile carriers are touting new cheaper wireless data plans, which actually make a ton of sense for those people who buy a smartphone with no intent of actually getting online with it. And there are a lot of people for whom that is the case. The early adopters of smartphones were IT guys who used the data like crazy. Now that the common folk carry smartphones, they don't use them to their full potential. It's still seen as odd when someone looks up information on the internet using their phone whether during a class or an informal conversation. Oh, who's Mr. Smart Pants, showing off his fancy phone? Well, you could have looked it up on your phone, too, but you apparently don't know how.

So there are some heavy users and some light users of mobile internet, so it makes sense to let the light users pay less, right? Have we learned nothing? Notice how more phones are coming with videoconferencing hardware and applications? Notice how everyone has downloadable apps for everything? Notice how more and more devices will act as a GPS with turn by turn directions? Notice how more content providers are advertising that their content is being optimized for these mobile devices? Does this sound familiar? There is more and better multimedia content coming available for smartphones, not to mention tablets which have finally started to take hold. These applications and content are user friendly enough that the non-techie masses will start using them more. The social stigma of actually using your device in a way that approaches its potential will decrease.

Let me put it more directly. Unlimited data plans on mobile devices served well to stimulate the generation of more multimedia mobile content. Now that there is plenty of content, service providers are dropping the unlimited data plans before everyone starts using the data fully, telling them that it is for their own good, but knowing full well that very soon everyone will be using more data and start slamming up against massive overage charges. Just like traditional landline high speed internet only a short time ago.

No More Books

It's interesting how an initiative touted by one group as being a revolutionary improvement will often be seen as counterproductive or ineffective by another group.

At a recent faculty meeting, where discussion of a new writing center was taking place, I was witness to such an example. One faculty member pointed out that as important as it is to help students improve their writing, which the writing center is intended to do, there is unfortunately something else that has the potential to hurt our students even more.

What's the insidious plague about to be unleashed on our students' writing skills? It is an initiative aimed at reducing the amount they have to read. If we require them to read less, it will have direct impact on their ability to write. Well, of course, the matchup between reading and writing seems pretty obvious, but why would anyone propose to dramatically reduce student reading?

It turns out the faculty member misunderstood the initiative. The proposal is to create integrated learning environments where students can access multimedia, direct themselves to activities based on formative quizzes, oh yeah, and read. His understanding was that the multimedia and pretty websites were going to replace books.

While it's true that the goal is to move away from traditional printed textbooks, the same text from the publishers is going to be integrated in an electronic form with the bells and whistles that improve student engagement and track progress in ways that a plain old book could never do. I know a lot of publishers create companion websites where students can access some bonus materials, but they're not integrated with the text. Bringing them together in one place reduces the extraneous cognitive load associated with having the text and bonus features in completely separate mediums, giving them more time and mental capacity to devote to their reading.

Trolling

Sitting in on some meetings with faculty members from several different departments, the discussion turned at one point to staying on top of student opinions, experiences, and problems they might be having.

photo by sue_elias
Various methods of keeping tabs on the student experience were discussed. One in particular was worthy of note. A faculty member shared her habit of trolling the course message boards and encouraged others to do the same. (Just to be clear, these are message boards open to anyone working on a given course, not locked down to just one section of a course inside an LMS.)

I alternated between staring in disbelief and suppressing laughter as I pictured a seemingly polite and professional faculty member taunting students in the official university message boards. I could understand if she wanted to do it under a pseudonym on a third party site. I mean, who doesn't do that, right? Right?

Obviously the behavior she engages in is lurking, not trolling. As nefarious as it sounds to be a lurker, it's really no big deal. It's just someone who sits by and watches the conversation but doesn't contribute. The real life equivalent of lurking is somewhat awkward; just sitting around watching people and listening in on their conversations is not considered polite. In the online world, however, lurking is often recommended, primarily as a way to get acquainted with a community before actively contributing in order to get a handle on the norms of that particular group and to get to know who the trolls are.

That brings us to trolling. It really doesn't have anything to do with trolls, as nasty as they are. Rather, it comes from a fishing term describing a method of dragging a lure behind a boat. The lures are often designed to mimic dying or injured fish. The idea is the troll tosses the bait out there and drags it around looking for easy prey. The victim thinks they're attacking a helpless victim, but they thus become the victim as they take the bait and are unwittingly dragged into an argument. Getting into an argument with a troll means you have lost. You are the victim. There is no way to win. The argument is what the troll is looking for. The flame war is the desired end result. Even the most glaring hole in a troll's argument cannot be addressed, because it is often simply more bait designed to keep the "conversation" going.

Chaos is only avoided by ignoring the troll or calling him or her out directly as a troll and then proceeding to ignore. I recently watched a video of a dude hitting himself in the face with a racquetball. In the comments, a troll starts an argument by claiming that the sport they were playing was squash, not racquetball. It's clearly racquetball. No logic prevails, as any evidence is simply countered by "it's squash". At one point, the troll even directly admits to being a troll and then immediately parrots back the line again, "it's squash" and the argument continues.

What's the point? Why would someone do that? It doesn't matter. Analyzing why someone would do such a thing is beside the point. The only thing that really matters is recognizing it and disengaging immediately.

Now imagine with me again the kind-hearted faculty member leaving our discussion on how to best engage with and assist our students, pulling up her laptop in some dark corner of the hallway, and leaving provoking comments on the student message boards.

A Whole New World

Last month, I mentioned how sometimes people will fight against something and by so doing cause what they were trying to avoid in the first place. Here is a similar example, this time of people arguing against something and thus making the point they were arguing against.



At East High School in Salt Lake City (yes, THE East High School), they are putting on a bilingual production of Aladdin. I think it's a great idea. Basically, half the school is Caucasian, one quarter Hispanic, and the rest other minorities. The problem is that only the white kids try out for the theater, until now.

They are coming together. They are all happy to have met people they didn't know before. They're helping each other with their languages. What a great story, too. Aladdin is all about not trying to be something you're not but loving others for what they are. Maybe next year they'll do a bilingual Romeo and Juliet.

The point I made in the first paragraph comes from the comments on the story. The more people complain about this production, the more their own xenophobia is apparent, thus reinforcing the need for such a production. Most of the first comments were people saying this was a bad thing, though it is balancing out with supporting comments as the day goes on. Some commenters claim this is a ploy to encourage acceptance of illegal aliens. One asks that we stop catering to minorities and accept each other without bringing race and culture into the conversation. Another asserts that a bilingual production won't help the Hispanic kids learn English, which is the global language of business and science. Two commenters unfold the logical fallacy that if we do things in Spanish, we'll have to start doing things in many other languages as well.

First, who knows if the Hispanic kids are illegal or not? Second, who cares? They may very well have been born here, and whether they were or were not, they likely have little choice but to live where their parents are. These are children we're talking about. How messed up is the request that we accept each other without talking about our races and cultures? If the acceptance only goes one way, from minority to majority, that is actually a request that minorities reject their own race and culture with an out where the majority doesn't have to accept anyone that's different.

When it comes to learning languages, I don't think we need to worry about whether or not the Hispanic kids are learning English. They've grown up here; they know the language. They weren't ignoring the theater because they don't know the language but because there was simply a cultural barrier that needed to be broken. It has now been broken.

It's the white kids who are less likely to know another language, like the joke (not actually a joke) that if you know two languages you're bilingual, three languages you're trilingual, and one language you're American. Learning a second (or third) language helps one understand his or own language better. And talk about the language of science being English, except that many of the terms we use come from Latin, upon which Spanish is based. A firm grasp of a Romance language makes Latin a lot easier.

I think it would be great if the slippery slope argument came true when it was put forward that if we allow Spanish, we'd have to do things in countless other languages. Okay, let's do it then. Let's mix in Arabic, Hindi, Mandarin, and Russian. We're still arguing about Spanish, when there are all these other languages that millions (even billions) of other people are speaking. We're not at risk of having the issues that China and India have with hundreds of languages swirled all over the place causing communication issues. English is already common, in spite of how poor of a language it is; we just need to add on a few more to spice things up.

The bigger question is one of culture, not language. Language happens to often be the piece that starts someone off into a better understanding of other cultures. Is there a better way to build cultural understanding than our children sharing language and song in a production about not trying to change yourself to impress others?

Improving Teaching by Understanding How People Learn

A colleague recommended a book he's been reading, “How People Learn: Brain, Mind, Experience, and School” by the National Research Council.

In the introductory chapter, they give some important concepts for teaching based on an understanding of how people learn. The main points:

• student initial knowledge and conceptions provide a foundation for further learning


• frequent formative assessment encourages metacognition and leads to deeper learning


• need to teach education students to help their students recognize and replace incorrect preconceptions


• in-depth coverage of fewer topics is better than superficial coverage in many topics


• to be an effective teacher, you need to have done in-depth study yourself


• assessment must test at the level teaching occurs, that is deep testing for deep teaching (students will do poorly on superficial testing if they've been taught for deep understanding)


• metacognition needs to be planned into the curriculum

The whole book is graciously provided online for free. See the complete table of contents.

Student Government or Start a School?

Someone has to plan parties. Otherwise they either wouldn't happen or would be very poorly organized. So there is a place for student government in college, but it should be renamed to reflect that limitation.

High school is all about being popular, so it makes sense that student government there would be structured around popular kids maintaining their popularity monopoly. It just doesn't translate to the college level.

College is a great equalizer. It's a time to regress to the mean. Ultra popular kids will find they really aren't that much more special than everyone else (big fish in little pond becomes little fish in big pond). Shy kids will break out of their shell. All that is really needed at this point is someone to throw a few big university-wide parties and several small department-sized parties to bring everyone together.

A recent article in the paper at USU regarding approval ratings of various facets of student government there focuses mainly on the fact that approval ratings are up, because they put on some great concerts this year. That's as it should be.

Just a couple months ago, there was another article about how student government was restructuring compensation for their officers, since as of halfway through the school year, they had put themselves $80,000 in the hole. I'm okay with someone who's bad at math putting on a big party, because the worst they're likely to do is go over budget by a few thousand dollars or run out of ice cream halfway through the party. The problem is when they start stepping out of their party zone to push for new buildings or try to influence academic programs, both of which they've also been doing over the past couple years. Major capital investments and toying with academics are just something they shouldn't be allowed to play with.

I'm not saying that there are no students that could contribute in a meaningful way to the university community or even do awesome things around the world. There are many who can and do. The people who will do these amazing things, however, are not likely to run for office, since they recognize its party-planner role. My dad was Business Senator about 40 years ago, so no offense, Padre, if you're reading this; maybe leave a comment if things were much different back then than they are now. From my own experience about 10 years ago as VP of one of the chapters of ACM on campus, we did get decent turnout to some great educational sessions on a variety of topics, but the billiards and pizza party at the end of the year got a lot bigger turnout than the demos on Flash or ASP.

To drive the point home, check out this video by current president Tyler Tolson, encouraging students to run for office. His point is basically that you get paid to eat dinner with university administrators and students follow you around campus (unless their shoe comes off like the dude on the far left at 1:01).


Compare that to Casey Allred, also an undergraduate student, who has created a non-profit organization, Effect International, and built a school in India with more on the way.


What do you do in your spare time? Throw a party? Rid the world of illiteracy? Choose what works for you, but be honest about it.

Problems in Student Teaching

This week is WGU's semi-annual academic meetings. We're almost getting too big for the conference center we've been meeting at, with about 700 people attending.

Today one of the presenters talked about a particularly difficult problem facing the teacher education programs. An important component of licensure is student teaching, and true to form, WGU has a different term for it: demonstration teaching. Regardless of what you call it, placing students in schools is becoming more difficult, and I imagine it's the same for schools across the country.

With the economy down, people are out of work, so they go back to school to either be better positioned to find a job now or to at least find a better job when the big recovery happens, so enrollments are up. Budgets in the states are down, so teachers are laid off, meaning there are fewer classrooms in which the increased number of teacher candidates can be placed.

Add in the pressures of NCLB, and a whole new challenge appears. In schools that are struggling to maintain AYP, teachers may decline to take on a student teacher because of the extra burden it imposes and the chance it could cause problems for their students' scores. For failing schools, well, it's not really an option to student teach at a failing school.

Unfortunately, I don't know what the solutions are. A change to NCLB seems like a good place to start.

Know Your Bill of Rights

While it is rare these days to have to assert one's rights as guaranteed by the Third Amendment, it's probably important to know what's in it. This is mainly to reduce the extent to which you look like a fool when you mix it up with one of the more commonly cited amendments.

Sending Mixed Signals?

We're heading up to go skiing at Beaver Mountain with the family tonight, which is always fun. Starting to get ready reminded me of the mixed signals Utah State sends when it comes to winter activities on campus. I love sledding on Old Main Hill, and ice blocking in the summer is also fun.

The student association lists sledding down the hill as an important campus tradition.

Then there's this video put out by the university highlighting the winter recreation close by campus. The whole video is great. Jump to 2:38 for video of people sledding on Old Main, and at 3:01, it blows me away what the dude does while skiing down the hill.



Just don't tell the risk management office what you're doing or you'll be severely warned about how dangerous it is.

I just wonder which way they want it. Is it a dangerous activity that the university should warn us to avoid or an advertisement to attract students to USU? I, for one, am receiving mixed signals.