Tag: learning

Things I Know 273 of 365: Value added isn’t

Value-added assessment is a new way of analyzing test data that can measure teaching and learning. Based on a review of students’ test score gains from previous grades, researchers can predict the amount of growth those students are likely to make in a given year. Thus, value-added assessment can show whether particular students – those taking a certain Algebra class, say – have made the expected amount of progress, have made less progress than expected, or have been stretched beyond what they could reasonably be expected to achieve.

– The Center for Greater Philadelphia

Professor Andrew Ho came and spoke to my school reform class tonight about the idea of value added and its space in the conversation on American education.

We started looking at a scatterplot of local restaurants situated by their Zagat rating and the Zagat average price per meal.

Ho then plotted a regression line through the scatterplot and took note of one restaurant that had a higher score than predicted for it’s cost.

The temptation was to claim our overachieving restaurant was a good buy for the money. Who’d expect a restaurant with such inexpensive food to have such a high rating?

Then he asked us what we didn’t see.

Portions, ambiance, quality, location, service, selection, etc.

Any of these is familiar to someone who’s debated with a group of friends when attempting to select a restaurant.

His point was simple. Expectations changes based on what you base expectations on.

Ho relabeled the axes – this year’s test results, previous year’s test results.

He asked us what we didn’t see.

Content, delivery, socioeconomic status, race, home life, sports, after-school activities, tutoring, mentoring, etc.

This is to say nothing of the fact that perhaps there is a natural spread to knowledge and growth that is beyond the influence of a teacher or the fact that different combinations of teachers in the life of a student in a given year could have varying effects on achievement.

A psychometrician, statistician and policy researcher, Ho then laid some data on us from the research on value added:

  • Estimates of value added are unstable across models, courses that teacher might teach, and years.
  • Across different value-added models, teacher effect. ratings differ by at least 1 decile for 56%-80% of teachers and by at least 3 deciles for 0%-14% of teachers (this is reassuring).
  • Across courses taught, between 39% and 54% of teachers differ by at least 3 deciles.
  • Across years, between 19% and 41% of teachers differ by at least 3 deciles.

He then made a point that’s come up time and again in my statistics course, “Any test measures, at best, a representative sample of the target domain.”

But we’re not seeing samples that are representative. According to Ho, “In practice, it is an unrepresentative sample that skews heavily toward the quickly and cheaply measurable.” We’re not learning about the population. Put differently, we can’t know all that we want to know. Anyone who says differently is selling something.

When questioned on teacher assessment in his recent Twitter Town Hall, Sec. Duncan said he favored multiple forms of assessment in gauging teacher effectiveness. Nominally, Ho explained, this makes sense, but in effect it can have unintended negative consequences.

Here too, Ho cautioned against the current trend. Yes, value added is often used in concert with observation data or other similar measures. If those observations are counted as “meets expectations” or “does not meet expectations” and all teachers meet expectations, though, we have a problem. The effect is to mute the impact of this measure in the composite. While it may be nominally weighted at 50%, if value added is the only aspect of the composite accounting for variance, “the contribution of these measures is usually much higher than reported, as teacher effectiveness ratings discriminate much better (effective weights) than other ratings.”

Ho’s stated goal was to demystify value added. In that he succeeded.

He left us with his two concerns:

  • The current incentive structures are so obviously flawed, and the mechanisms for detecting and discouraging unintended responses to incentives are not in place.
  • The simplifying assumptions driving “value added,” including a dramatic overconfidence about the scope of defensible applications of educational tests (“numbers is numbers!”), will lead to a slippery slope toward less and less defensible accountability models.

I’d hate to think we’re more comprehensive in our selection of restaurants than teacher assessment.

Things I Know 265 of 365: I’ve been worrying over grades

But there are advantages to being elected President. The day after I was elected, I had my high school grades classified Top Secret.

– Pres. Ronald Reagan

I’ve decided, if possible, to take all of next semester’s courses pass/fail.

It was a decision I almost made when I registered this semester, but the schooly devil on my shoulder shouted pretty loudly.

I’m not failing any of my classes and I’m reading and learning more than any aggregate moment of my undergraduate career. The problem is that I’m worried about the grades in a way that makes me uncomfortable and that leaves me wondering if the learning I felt like I was doing matched the grades on my assignments.

This problem works both ways.

Earlier this week, I received a graded stats assignment (and you know how much I’m loving stats). Along with the comments from the Teaching Fellow (what Harvard calls TAs) was an A. I received an A on the assignment.

Then I got angry. I’ve been reminding myself any B I’ve received this semester was someone else’s interpretation of my learning and not a reflection of what I’d actually learned on the assignment. Most of the time, I’ve interacted with the grading TF no more than a sum of 10 minutes. Even if it’s been more, the samples of my work and thinking my graders have seen have been minimal. It’s a little like a standardized testing window.

My anger at the A rested in how quickly I was willing to accept a complimentary grade when it validated my self concept.

I can’t have it both ways. No matter my reaction, the effect is the same. Grades distract me from learning.

This is not to say, as Dave Thomer commented the other day, that I don’t respect and internalized my teachers’ critiques of my work. I’m here to study with experts and learn from them. Part of that means submitting my work for their response.

Whereas a grade hits me like a period of exclamation point marking the end of my thinking on the matter, a paper returned riddled with questions and comments begs a conversation.

I read a grade as, “We’re done here.”

I read comments as, “Say more.”

One of these is internalized as a statement of worth.

The other is read as the invitation to keep thinking and asking questions.

I’m hoping removing overall grades will cancel out some of the background noise and help me focus on my learning and my professors’ coaching of that learning.

Things I Know 262 of 365: I wasn’t paying attention

Our inventions are wont to be pretty toys, which distract our attention from serious things. They are but improved means to an unimproved end.

– Henry David Thoreau

Yesterday, I was the student I’ve heard teachers worry about as they describe the detrimental effects of lifting prohibitions on mobile devices in the classroom.

Two-thirds of the way through 3-hour class, in class discussion, someone made an interesting point I hadn’t before considered. The point wasn’t made directly to me or even in my general discussion. Within the lecture hall, the content of the remark was diluted by follow-up comments moments after it escaped my classmate’s mouth.

I’m not sure what the follow-up comments were. The initial remark made me curious and I started following my questions around the Internet. Browser tabs began propagating one another, and I soon had enough information to qualify as deeper understanding.

I knew something about what my classmate said that I hadn’t known before I started poking around online.

I hadn’t followed or heard a single word of the discussion once I’d tuned out; yet, I’d kept on learning.

What’s more, once I returned to the conversation, I was able to bring more information with me and deepen the level of discourse.

I’ll admit to feeling a little guilty for tuning out. The conditioned student in me worried the professor would notice I wasn’t following the conversation and be upset or cold call me to bring my attention back.

That’s where my dissonance lived.

I was learning, but I wasn’t in the conversation.

I was on topic, but I wasn’t in sync with those around me.

After class, the whole thing stuck with me. The kind of augmented learning I did isn’t possible or even allowed in most classrooms. Teachers and administrators are worried access to tools and information will lead to distraction in the classroom. I’ve taught long enough with cell phones and instant messenger in my classroom to know there’s some truth to that.

The experience also taught me the need to shift my game. If I was doing things or sharing information students’ points of access could provide, I was wasting my time. Constant information access meant I could focus on what we could do and build with that information. It meant we were free to ask better questions.

That’s what I did last night, deeply engaged in my learning and worried those around me would think ill of me for asking questions to build the fund of knowledge.

Not paying attention, as it turned out, didn’t mean I wasn’t learning.

Things I Know 260 of 365: I’m not sure what I did right

When we fail in this diagnostic role we begin to worry about ‘assessment.’

– David Hawkins

I’m struggling to write tonight. I’ve been struggling to write for the last few days.

I’ve an assignment due tomorrow – 8-10 pages, and I can’t get myself invested in it. Or, I’m too invested in it.

For the last assignment in this class, I submitted work of which I was proud. I spent time and thought on the assignment. I worked to refine my thinking and understand which other thinkers served as progenitors to my ideas.

My work was submitted with a feeling of having been thoughtful and diligent in my work. I had learned something new and refined   old thinking.

When I got my assignment back, I struggled to find positive comments. I struggled to find comments that were in response to my ideas.

I didn’t need praise lobbed at me or ego stroking. I just needed a clear sign of where I was on the right track; otherwise, I start to question if I was anywhere near that track.

Because I am who I am, I submitted a re-write of the assignment. Re-doubling my efforts, I consulted the rubric even more the second time than the first.

While my grade on the second attempt was higher than the grade on the first, I’m still sitting here stymied as I work to complete this new assignment.

It’s a horrible feeling.

I don’t know what I did well in the last assignment upon which I can build for this go-round. I have lists of things to avoid, but I don’t know what I’m good at in context of trying to do what’s been asked of me.

I’ll write more tonight.

I’ll write more tomorrow.

I’ll turn in my assignment tomorrow.

I’ll be hesitant to feel proud.

And the thing that kills me – that absolutely drive me batty – the work I did on the first assignment and the work I did for the re-write was fine work. I am still proud of that work.

But there’s a teacher’s opinion in there. There’s a teacher’s opinion muddying the waters of my learning.

And I’m really hating the fact that matters to me.

Things I Know 253 of 365: I chose this frustration…and it’s excellent

Pessimist: One who, when he has the choice of two evils, chooses both.

– Oscar Wilde

Part of the beauty of the program of study on which I have embarked is its lack of requirements. Though there are some sorts of courses I need to take to graduate from my particular program, I’ve much latitude in deciding exactly what that looks like. This semester, I’ve chosen to fulfill four of the five requirements.

The idea was to open the spring up to cross-registering courses in the Kennedy School or Harvard’s Business School or School of Public Health.

As I registered at the top of the semester, the freedom of choice I told myself I’d experience in the second half of my time here drove a good deal of the selections I made.

With a certain degree of surprise, my statistics course continues to be the course in which I most feel myself and my understanding of my capabilities growing.

It is also the course in which I know I’m making the most mistakes. I simply don’t know a lot of what we’re learning about. As such, I tend to misuse the language of statistics. It’s like someone who’s fluent in Spanish visiting Paris and recognizing just enough of what’s being said to make the inability to communicate perfectly frustrating.

As I sat in the library tonight compiling a report that referenced t-tests and chi square tests, friends and fellow classrmates happened by. They noticed the sprawl of papers covering my study carrel and commented they were glad they weren’t in the course. One person even said you couldn’t force her to take the class.

It occurred to me then that this might be why I’m enjoying my statistics class so much. No one made me take it. It is a pre-requisite for the next level of stats in the Spring, but I’m not taking that class. None of the millions of possible next jobs after school requires me to have a knowledge of statistical analysis.

I’m enrolled in the class because it seemed like it would be interesting and I didn’t know anything about the subject matter. It is new.

Each time a homework assignment makes me want to disrupt the tranquility of the library with a yelp of, “For the love of all that is holy, someone just tell me the answer,” I remind myself – I chose this. No external, deus-ex-machina force worked to force me into this class. I chose to learn this, to work with material heretofore unknown to me.

Having that choice and agency have made all the difference. I am learning because I chose what to learn. I was curious and free to follow that curiosity.

Things I Know 252 of 365: How we shape learning shapes learning

I’ve been thinking about shape a lot lately. Specifically, how the way we shape things shapes how we shape thoughts. I know I’m not the only one to have considered this, but I’m the only one in my head to be considering it, so I’m going with it.

Friday, I visited an elementary school to observe and report.

I saw first, second, fourth and fifth grade classrooms.

In working toward a goal for a learning task in one of my classes, I noted the arrangement of each of the classrooms. My sketches weren’t perfect, but they reflected the general arrangement of each room.

In first grade, there were groupings of five or six tables, there was a carpet by a dry erase board. A teacher desk suffocated beneath papers. A kidney-shaped and a circular table both hinted at where the students might work to complete a collaborative task or work together with the teacher’s help. When I walked in, students were everywhere. Some were at their desks completing math work. Some were reading. Others were working together on the ground to paint what looked like it was the makings of a tree trunk. At one point, students transitioned from their myriad tasks to a whole-class reminder of the previous day’s learning and then community time at the carpet with the teacher.

The second grade class had even more of the frenetic energy you’d hope to see in a place where people are learning. The class’s co-teachers were across the room from one another working with small groups of students in rotation while the other students worked their way through stations where they read, counted using number lines, colored, completed their poetry journals and fitted blocks together to form vocabulary words. Just when I thought I’d gotten a handle where everyone was, they slid seamlessly to another station.

The portion of fourth grade I observed had fewer stations, but the co-teachers worked together to move student learning. One sat at a kidney-shaped table with a small group while the larger class worked on an assignment in organizations ranging from 1 to 6. The task involved manipulatives and the students each used them to find answers to the problems they were addressing and explain their answers to group members who weren’t seeing their logic. Though focused on one task, the room was still abuzz with difference.

Fifth grade took a turn. Groupings of desks changed from a standard of 4-5 to 3-4. The room had a clear front and back. The teacher was at the front. Her desk was at the back. The students were facing her. Focused on a singular task, student shared their answers and the teacher asked if the class thought those answers were correct.

I’ve only got a sampling of four classrooms, but I think I can see where this is going. All I need do is examine the learning spaces I head to throughout the week to see the natural end of this progression.

Each class is a variation on a theme. Scaled up and down according to the room and how many people we need fit inside it. The these horseshoes are where we learn about reforming how students learn. They are where we read about and discuss the importance of collaboration and choice. In these spaces, we examine student- versus teacher-centered practices and question why it is so difficult to move teachers’ practices to the former.

Some professors have attempted to break the space against itself and encouraged group work and movement. But the spaces weren’t meant for this. They don’t invite creative uses.

I looked at the collection of how teachers were using the spaces in the schools I’ve visited this year and noticed a trend.

The learning was different. The lessons were different. The voices and sizes were different. But the spaces moved toward one singular design.

I know where this leads my thinking, and I wonder what kind of thinkers, creators and citizens these spaces encourage and invite. No matter our professed values, are we building spaces that ask students to question, build and move forward?

Some Useful Words from Ted Sizer on a Common Curriculum

From Horace’s Compromise:

Some today, with earnest good intentions, urge that a common core of subjects be legislated for high school students. Depending on one’s point of view, much of this certainly is nice. Laudable or not in the abstract, however, if it is mandatory, it is an abuse of state power, an excessive reach of political authority. Again, the state is fully justified in providing it at public expense, if it wishes, and prescribing it for certain certificates and diplomas that citizens may voluntarily choose to earn.

Some others say that an adolescent should have a “high school experience,” something that is inherently a Good Thing, an experience that teaches young people to “get along with others.” Proponents of this view offer no evidence for support of their argument for mandatory “residence” at school. This is prudent on their part: there isn’t any. Most real reasons for enforced attendance actually turn on the need to preserve adults’ jobs. Compulsory attendance in an educational institution should cease when a young citizen demonstrates mastery of the minima, and most young citizens should master those minima before senior high school. As a result, schooling for most adolescents would be voluntary. Few would be compelled to attend high school, though a prudent state would vigorously encourage it. High school would be an opportunity, not an obligation.

Things I Know 246 of 365: I’m deciding to learn something new

True genius resides in the capacity for evaluation of uncertain, hazardous, and conflicting information.

– Winston Churchill

In my first weeks at SLA, I got a little terrified. I inherited my classroom midway through the first quarter. By the time I’d started thinking about getting my bearings another teacher approached me, “What are you planning for your benchmark this quarter?”
For what was the fiftieth time since joining the school, my heart stopped.
I had no idea.
I talked to Chris. It was a moment of intellectual cardiopulmonary resuscitation.
Essays, he explained, could be projects.
I breathed again.
I can do essays. They are the coin of the realm for the bulk of schooling. Whether you believe the 5-paragraph essay is the devil or the first step to clean livin’, schools find ways to get kids writing essays. Take a look at standardized test scores and you won’t be able to turn around without bumping into a schools whose writing scores outpace reading, math and science.
I’ve got three major essays in the offing here at school. This is on top of the weekly essay assignments for two classes. Add to those the daily postings here, and I start to feel like the essay king.
It’s ok. Essays are my sport.
So, I’ve decided to take up a new sport – one I appreciate watching, but have no idea how to play.
I’m going to learn to create data visualizations.
In google reader, my “Infographics” folder is my favorite.
I’ve been quietly building my collection of resources in delicious.
After I complete my next two essays, I’m starting. Seriously.
As many people better versed in the visualization of data have written, information and making sense of it are the coins of the realm for the modern age.
This realization is my secondary drive. Most of all, I’m curious. It’s the same things that led me to open up and dissect every telephone I could get my hands on as a little kid. It’s what prompted me to mix rain water, onion grass and other things I found in my yard and leave them in the garage to see what happened.
I’m curious about something I don’t know. So, I’m learning a new sport.

Things I Know 244 of 365: We only need to half-flip the classroom

Chatting with a friend today, I explained the premise of the flipped classroom:

1. Teacher makes videos of shortened versions of lectures.

2. Students watch lectures at home.

3. In the physically shared space of the classroom, the community practices at the learning.

I think I’ve got a way to make the whole experience better.

Stop making the videos.

I hesitate to write this. The flipped classroom is as close as we’ve come in a long time to an institutionally-backed shift from teacher-centered to student-centered classroom practice. The mastery system is an improvement from the traditional way of doing things. The model frees teachers to provide students with individual attention. These are good things.

Part of me wants to say, “Keep the videos so long as it transforms classrooms to studios, labs, workshops and playgrounds of learning.” But there’s a way to make less work for teachers and students in this equation:

1. In the physically shared space of the classroom, the community practices at the learning.

The Internet is replete with videos, how-tos and step-by-steps explaining almost any lesson a teacher could conjure. What’s more, many of these resources are better than what a typical teacher has time to create.

Some tips for a half-flipped classroom:

  • Use diigo, stumble upon, delicious or another social bookmarking tool to collect any and every resource students find in connection to the learning they’re engaging in at the moment. Come up with a class tag, unit tag, lesson tag and challenge students to find the resources that make the learning work best for them.
  • Give time in class to talk about what they’ve found and how they found it.
  • Have a class space for the curation of content. It doesn’t have to, and shouldn’t be, any one kind of space. Wiki? Great. Google site? Tremendous.
  • Be available and encourage student availability. For me, this meant creating a google voice number that fed student text messages to my e-mail account, being available through Facebook, twitter and IM. For anyone else, it might mean any one of these or something else.
  • Learn along. Nobody likes a know-it-all, but everyone likes to know it all. Any chance I had to learn along with my students, I took it. They knew more about more than I did. I knew literature, grammar and writing. That’s what I brought to the room. From there, I was genuinely curious to learn what they knew – not from an assessment standpoint, but from a learning standpoint.

I’ve two other arguments against the fully flipped classroom. They are the natural derivatives of the Law of Unintended Consequences. First, we’ve taken enough of our students’ time already. Though our hours or 45-minutes with them at a time might seem always too short, they experience a school day full of these bursts. Giving them more to do “for us” won’t make our classes more important. They’ll merely seem more urgent. Play is an endangered species. Let’s respect the ecosystems of our kids’ lives.

My second argument against full flipping is that we’re fooling ourselves if we think our students will continue to watch these videos over years. At some point, the novelty will wear off. The Freakonomics folks posted today about the Indian government’s issuance of masks to workers in the field who were in danger of tiger attacks.

Because tigers attack from the rear, workers wore the masks backwards to fool the tigers.

It worked – for a while. Tigers have started to learn the masks are just that.

Rather than masking students’ experiences in the novel, let’s outfit the experiences with the authentic.

Things I Know 240 of 365: I wrote with the world

The world and I wrote a paper Friday.

By midnight tonight, I’m to submit my Theory of Learning for A-341 Supporting Teachers for Instructional Improvement. I’d been resisting the writing of the paper. After railing against the silver-bullet approach to education, sitting down to distill my beliefs into a single theory lived in a hypocritical room of my brain.

The temptation was strong to submit a Word doc containing only a link to this space, but that steps outside the bounds of the assignment requirements.

Two weeks ago, I asked 5 people to take a look at the first few pages of a rough draft of the paper. I’d written it up in Google Docs and shared it out.

Friday, I needed to get down to business. I wasn’t going to face a long weekend with an assignment hanging over my head the entire time.

I sent out this tweet and started writing:

Before long, other folks from wherever had jumped into the doc and started lurking. A few left comments on my friends’ comments. My friends, either from the doc or via e-mail, responded to the comments.

I kept typing.

Dan Callahan, who’s about as fine a teacher and person as you’re likely to meet, retweeted:

Google Docs let me know as more people joined me in the doc.

I kept typing.

As I neared the end, this message popped up in the doc’s chat window:

On the other side of the world, a teacher I didn’t know was reading my thinking as I cobbled thoughts together. Even more, she was moved to interact. We talked about our experiences in modeling and eliciting passion from students and shared a bit about our backgrounds. I learned her name is Jo:

I told her the doc would remain live as long as Google let it be so and that the copy would be posted here. I offered to brainstorm with her and her teaching partner if they’d like – to continue connecting.

And then she left.

I kept typing.

The difference at that point was huge.

I’d been putting together a theory of learning based on the ideas that:

  • Students learn best when they are in an ethic of care.
  • Students learn best when they know something about what they are learning.
  • Students learn best when the learning situation has real stakes and is challenging.
  • Students learn best when the learning is playful.

I’d been professing all of this to complete an assignment that initially spoke only to the second tenet. I knew a little bit of where I spoke.

The rest, as a student, I created.

As soon as I invited my friends, those whose minds and passions inform my thinking, I chose to surround myself in an ethic of care. In the initial stages of the rough draft, my sister Rachel watched from Missouri as I typed in Somerville. She offered encouragement and asked prodding questions. What I was saying mattered to someone other than me.

Each time Bud or Ben or Debbie pushed back, my learning was more playful. Every comment in the spirit of “What about X?” was an intellectual chess move asking me to refine my process and play with my thinking more deeply.

As soon as Jo entered the chat and asked if she could use a piece of thinking that was being created as she typed, the stakes became real for me. What was otherwise to languish as another artifact of academia destined for the eyes of a professor and teaching assistant was transformed into a guide of practice that would, in some way, affect the learning of children half a world away.

Unless a teacher is completely out of touch with his students, an assignment is likely to connect to students’ previous learning and fulfill my second tenet.

The other three, though, they take work. I write this as a teacher and a student – that work makes all the difference.