Tag: statistics

8/365 ‘Right’ Answers are Overrated

As I work my way through Duckworth, I’m tempted to temporarily change the name of this blog to Reading So You Don’t Have To.

I threw the picture below up on instagram as I was reading last night, and feel like it needs a more prominent display:

A Thought on Assessment

 

My comment attached to the photo was something along the lines of this way of thinking being the only thing I needed to guide my thinking on assessment. That stands. As I continue exploring The Having of Wonderful Ideas, Duckworth is pushing my thinking on assessment even more. Actually, she’s not pushing my thinking so much as putting thoughts I’ve had before into better prose than I’ve yet managed.

It occurred to me, then, that of all the virtues related to intellectual functioning, the most passive is the virtue of knowing the right answer. Knowing the right answer requires no decisions, carries no risks, and makes no demands. It is automatic. It is thoughtless.

 

This past semester, for the quantitative methods course I was taking, we used a textbook of dubious pedagogy. At the end of each section, though, were some practice problems of the type I remember from my math textbooks of my youth.

Because statistics isn’t exactly where my innate intelligences lie, I found myself frequently stopping to attempt the practice problems. I was curious about these new ideas and this mostly new language of statistical reasoning. I filled large pieces of chart paper with my thinking on these problems with arrows and borderlines to delineate where one thought took a break and moved on to be another thought.

Not always did I arrive at the right answer. What I found, and what surprised me, was the sense of joy and accomplishment I felt when I had an answer and could explain those with whom I studied how I got to that answer. When the answer was wrong, being able to hold up the path I’d taken to reach it somehow took the sting out of its wrongness.

I wouldn’t have paused to appreciate and “meet” the thinking necessary to solve those problems if I simply knew the right answer. If it had been automatic and thoughtless as Duckworth describes, it also would have been a hollow victory if it had been any victory at all.

Things I Know 273 of 365: Value added isn’t

Value-added assessment is a new way of analyzing test data that can measure teaching and learning. Based on a review of students’ test score gains from previous grades, researchers can predict the amount of growth those students are likely to make in a given year. Thus, value-added assessment can show whether particular students – those taking a certain Algebra class, say – have made the expected amount of progress, have made less progress than expected, or have been stretched beyond what they could reasonably be expected to achieve.

– The Center for Greater Philadelphia

Professor Andrew Ho came and spoke to my school reform class tonight about the idea of value added and its space in the conversation on American education.

We started looking at a scatterplot of local restaurants situated by their Zagat rating and the Zagat average price per meal.

Ho then plotted a regression line through the scatterplot and took note of one restaurant that had a higher score than predicted for it’s cost.

The temptation was to claim our overachieving restaurant was a good buy for the money. Who’d expect a restaurant with such inexpensive food to have such a high rating?

Then he asked us what we didn’t see.

Portions, ambiance, quality, location, service, selection, etc.

Any of these is familiar to someone who’s debated with a group of friends when attempting to select a restaurant.

His point was simple. Expectations changes based on what you base expectations on.

Ho relabeled the axes – this year’s test results, previous year’s test results.

He asked us what we didn’t see.

Content, delivery, socioeconomic status, race, home life, sports, after-school activities, tutoring, mentoring, etc.

This is to say nothing of the fact that perhaps there is a natural spread to knowledge and growth that is beyond the influence of a teacher or the fact that different combinations of teachers in the life of a student in a given year could have varying effects on achievement.

A psychometrician, statistician and policy researcher, Ho then laid some data on us from the research on value added:

  • Estimates of value added are unstable across models, courses that teacher might teach, and years.
  • Across different value-added models, teacher effect. ratings differ by at least 1 decile for 56%-80% of teachers and by at least 3 deciles for 0%-14% of teachers (this is reassuring).
  • Across courses taught, between 39% and 54% of teachers differ by at least 3 deciles.
  • Across years, between 19% and 41% of teachers differ by at least 3 deciles.

He then made a point that’s come up time and again in my statistics course, “Any test measures, at best, a representative sample of the target domain.”

But we’re not seeing samples that are representative. According to Ho, “In practice, it is an unrepresentative sample that skews heavily toward the quickly and cheaply measurable.” We’re not learning about the population. Put differently, we can’t know all that we want to know. Anyone who says differently is selling something.

When questioned on teacher assessment in his recent Twitter Town Hall, Sec. Duncan said he favored multiple forms of assessment in gauging teacher effectiveness. Nominally, Ho explained, this makes sense, but in effect it can have unintended negative consequences.

Here too, Ho cautioned against the current trend. Yes, value added is often used in concert with observation data or other similar measures. If those observations are counted as “meets expectations” or “does not meet expectations” and all teachers meet expectations, though, we have a problem. The effect is to mute the impact of this measure in the composite. While it may be nominally weighted at 50%, if value added is the only aspect of the composite accounting for variance, “the contribution of these measures is usually much higher than reported, as teacher effectiveness ratings discriminate much better (effective weights) than other ratings.”

Ho’s stated goal was to demystify value added. In that he succeeded.

He left us with his two concerns:

  • The current incentive structures are so obviously flawed, and the mechanisms for detecting and discouraging unintended responses to incentives are not in place.
  • The simplifying assumptions driving “value added,” including a dramatic overconfidence about the scope of defensible applications of educational tests (“numbers is numbers!”), will lead to a slippery slope toward less and less defensible accountability models.

I’d hate to think we’re more comprehensive in our selection of restaurants than teacher assessment.

Things I Know 253 of 365: I chose this frustration…and it’s excellent

Pessimist: One who, when he has the choice of two evils, chooses both.

– Oscar Wilde

Part of the beauty of the program of study on which I have embarked is its lack of requirements. Though there are some sorts of courses I need to take to graduate from my particular program, I’ve much latitude in deciding exactly what that looks like. This semester, I’ve chosen to fulfill four of the five requirements.

The idea was to open the spring up to cross-registering courses in the Kennedy School or Harvard’s Business School or School of Public Health.

As I registered at the top of the semester, the freedom of choice I told myself I’d experience in the second half of my time here drove a good deal of the selections I made.

With a certain degree of surprise, my statistics course continues to be the course in which I most feel myself and my understanding of my capabilities growing.

It is also the course in which I know I’m making the most mistakes. I simply don’t know a lot of what we’re learning about. As such, I tend to misuse the language of statistics. It’s like someone who’s fluent in Spanish visiting Paris and recognizing just enough of what’s being said to make the inability to communicate perfectly frustrating.

As I sat in the library tonight compiling a report that referenced t-tests and chi square tests, friends and fellow classrmates happened by. They noticed the sprawl of papers covering my study carrel and commented they were glad they weren’t in the course. One person even said you couldn’t force her to take the class.

It occurred to me then that this might be why I’m enjoying my statistics class so much. No one made me take it. It is a pre-requisite for the next level of stats in the Spring, but I’m not taking that class. None of the millions of possible next jobs after school requires me to have a knowledge of statistical analysis.

I’m enrolled in the class because it seemed like it would be interesting and I didn’t know anything about the subject matter. It is new.

Each time a homework assignment makes me want to disrupt the tranquility of the library with a yelp of, “For the love of all that is holy, someone just tell me the answer,” I remind myself – I chose this. No external, deus-ex-machina force worked to force me into this class. I chose to learn this, to work with material heretofore unknown to me.

Having that choice and agency have made all the difference. I am learning because I chose what to learn. I was curious and free to follow that curiosity.

Things I Know 229 of 365: I’ve seen Problem-Based Learning from the other side

It takes half your life before you discover life is a do-it-yourself project.

– Napoleon Hill

I just turned in my second statistics assignment. I should note (and I’m sorry Mr. Curry), when I took statistics during undergrad it became a sad march toward intellectual self-destruction. I hesitate to say intellectual, but the professor certainly attempted to steer my thinking that direction.

More often, my thinking was, “How does this count as math? I know calculus. How is this math?”

It wasn’t pretty.

My current statistics professor came with glowing reviews – from everyone. Everyone.

And he’s fantastic.

A lecture hall can be a stuffy space.

A statistics course can be a stuffy space.

The intersection is potentially numbing.

Not with Terry Tivnan.

In a course explicitly designed with the beginner in mind, Professor Tivnan works to set a pace and climate that has yet to have me feeling out of my depth.

Given the laughter and applause that pepper our classes, I’d say my classmates are in a similar situation.

And then the assignment came.

Now, remember, I have been teaching in an inquiry-driven, project-based school for the last for years and another school for two years before that that was doing those things, but didn’t think to say so. Not only is this learning I believe in, it’s learning I’ve assigned as well.

Until recently, it hadn’t been learning I’d experienced. Seems appropriate I dove into the process in a field for which I’ve less natural predilection.

Without going too deeply into details, our assignment gave us two data sets, some information about national trends regarding that data, and asked us to compare the data and write up a report for a fictional school board regarding our findings.

That’s it. No one outlined steps. No one said this is the information you must report.

“How are these two things related, and what does that mean?” we were asked.

It hurt my brain.

A lot.

Unclear as to how to approach the problems and feeling the wait of my mathematical past, I avoided the assignment for as long as I could.

I worked to help classmates make sense of the work, while avoiding my own.

And then I realized what he had done.

He wanted us to own the process. I’ll get nowhere if I have to look to an authority each time I need to decide when and how to use a “z score” or the importance of a weighted mean. I needed to own it.

The process needed to be mine.

Now, these are things I’ve professed for years. I’ve stood in front of audiences and classrooms and argued the importance of this kind of learning.

Here’s the thing – it’s tough.

As incredibly difficult as shaping a lesson or unit plan for problem-based learning may be, learning that way is incredibly difficult.

From several classmates I heard cries of, “Why won’t he just tell us what he wants or what to do?”

I’d heard that before.

“But how do I do it, Mr. Chase?”

As supportive as I’d meant to be, I never truly understood the difficulty involved in adapting new habits of learning.

I expect it’ll get easier – not quickly – as we’re expected to do more on our own with the knowledge and understandings we’re acquiring.

For this go ‘rough, it was tough. I need to remember that.