Skip to Navigation
University of Pittsburgh
Print This Page Print this pages

October 27, 2005

Student data improves teaching, Harvard prof says

A little data can go a long way.

That was the lesson from a Harvard education professor and statistician who spoke here on “How Do Faculty Know When They Are Achieving Good Outcomes With Students?”

A decade ago, Richard J. Light, who is the Walter H. Gale Professor of Education and director of the Harvard Seminar on Assessment, was charged by Harvard’s president to oversee an analysis of the Ivy League school’s undergraduate educational programs. Among other sources of data, he and colleagues, which included faculty and academic advisers, developed a 55-question survey that would test freshman attitudes and expectations, with a follow-up survey for when they became seniors.

“We were asked to look at methods of teaching, advising and quality of life on campus: What are we doing well? What not so well?” said Light, who gave a presentation directed toward faculty here Oct. 14 at the fifth annual Symposium on Undergraduate Advising. The symposium is designed primarily to provide continuing education to Pitt student services specialists, particularly academic advisers, and to share knowledge across disciplines.

According to Light, the data gathered by the one-on-one, two-three hour interviews with undergrads led to a number of curricular and teaching methods improvements and policy changes.

Light was careful to preface his remarks by saying that his analysis applied specifically to Harvard, and may or may not be useful at other institutions.

“First I want to share a little anecdote,” Light said. “We met once a month for a year to come up with the questions. I noticed a table of eight science faculty from different science disciplines always sitting together. It made me curious about what their concerns were.”

The common thread of the science faculty members’ discussion was how to find the root causes (and perhaps the solution) for why such a large number of students, upon arrival, say they plan to major in a science, but end up majoring in something else, usually one of the humanities.

“About 35 percent of our entering freshmen say they’ll major in chemistry, but more than a third of them who say that don’t,” Light said.

“There are many possible reasons: the work is too hard, the teacher is a snore. These faculty were saying, ‘Nothing against psychology, English or history, but why am I losing these kids?’”

Also, while a high percentage of undergrads were flowing from sciences to humanities, the traffic was strictly one-way: “How many from humanities were switching to sciences? Three! Not 3 percent, but three out of 2,000!” Light said.

The science faculty group thought at last they had come up with a succinct way to get at this problem. “It was a four-word question they wanted on the survey: ‘How do you study?’” Light said. “My first reaction was: It’s a little vague, but they said, ‘We worked three months on crafting those words,’ so we kept it in.”

The very first time Light asked that question in an interview the student responded: “That’s a little vague. Could you re-word the question?”

“I could have, but I had to stop myself to preserve the reliability and validity of the statistics,” Light said. “It’s the nature of a questionnaire’s validity that you stick to the questions.”

At another interview, the student replied that how he studied depended on the subject. “We have it right there at the interview what courses the student is taking, so I thought it was all right to look at that list and say, ‘biology,’” Light continued.

“The student responded, ‘I study with my biology study group, of course. We work on the problems individually and then we meet once a week for about an hour and a half over dinner.’

“‘And your professors allow this?’ I asked, knowing that in my day it was considered cheating,” Light said.

The student said, “About half of my professors encourage it; others say, ‘Do it alone or it’s cheating.’ But by far the better way to learn is with a group, definitely.”

Light added: “He also reminded me that he still had to prove he knew the stuff on the exams.”

“Well, what’s the take-away? The more we looked into this the more we discovered. We learned that to study in groups increased student learning and that group discussions meant more student engagement. Now at Harvard every professor insists students study in groups and, since then, the persistence in sciences has crept higher up by about 2 percent a year,” Light said.

“’We also learned that men are much more comfortable studying in groups than women,” which may help explain why a higher proportion of men than women are sticking with the sciences, he added.

Light cited other examples where data had an effect on either curriculum or policies at Harvard College.

Faculty were asked which skill they would most like to see student improvement in over the four years, and the most frequent answer by a factor of three was writing skill, Light said.

“The take-away from that is that now Harvard requires a course in expository writing no matter the skill level of the student,” he said. “It’s the only required course at Harvard College.”

In order to measure whether student writing did improve during the undergraduate years, Light’s group designed a writing test that is scored by outside experts.

“At Harvard, most entering freshmen write well,” Light said. “Some don’t, of course. But we were looking at measuring outcomes when it comes to student writing. Why do some improve?”

So the entire entering freshman class is asked to take a one-hour ungraded writing test the day before classes start. They are given a small sample of Plato’s writing on Socrates and asked to write on: “If Socrates were alive, would he approve or disapprove of affirmative action?”

There is no correct answer; the point is to measure the writing skill level, Light said. “For entering freshmen, 7.2 was the average score out of 10.”

Three and a half years later, the same students were asked to write for an hour on: “If Lincoln were alive today, would he be a Democrat or a Republican?”

“Again, no right answer,” Light said. “For the near-graduates, the average score was 8.9, which is a highly statistically significant improvement. So that’s great, right?”

On the surface, yes, he said. But after drilling into the data Light discovered that humanities students improved a lot and social sciences students showed some improvement; but for students in the physical sciences, engineering, computer science and math there was no improvement.

“So we made the curricular change that everyone majoring in the sciences must take two courses of intensive writing,” Light said. “This was a concrete change that came directly from an outcome measure, one that was put in place at low cost: not free, but it did not cost a lot either.”

Harvard has an advantage over most universities in that it offers hundreds of very small classes, typically four-eight students in tutorials for upperclassmen.

“When we measure learning, we know small classes do better,” Light said. “But the shock is that some faculty use these classes to lecture. What a waste!”

That fact led to questions on what students felt was the best teaching method, he said. Overwhelmingly, students rated the best teaching method as “creating a debate in class.”

“Now, it’s fair to ask, how do you do that in calculus?” Light pointed out. “But in most subjects it’s easy to create a debate,” and faculty are taking this data to heart in their teaching, he said.

Light offered two other examples of how data gathered from student surveys can inform the educational process.

“At Harvard at least, there has been a dramatic change in the last 10 years in how students perceive the faculty and their teaching,” Light said. Today’s students do not want to know the faculty member’s politics, for example. In fact they resent it when a faculty member introduces his or her political opinions, he said.

That’s something that needed to be measured statistically, because it is not necessarily intuitive, Light pointed out.

In a second example, seniors were asked to name three people they admire most and why; they could name anyone from anywhere in the world, living or dead.

The top eight finishers on the list were in order: Mahatma Gandhi, Franklin D. Roosevelt, Ronald Reagan, Winston Churchill, Nelson Mandela, Margaret Thatcher, Martin Luther King and Anwar Sadat.

“One thing is the list is not American-centric; five of the eight are not American. Also, four are people of color,” Light pointed out. Moreover, the philosophies of the group are diverse and the students’ choices were not overly influenced by very recent history, he added.

“Frankly, I was thrilled,” Light said. “Maybe I wished there were more women on the list, but I thought it was an impressive mix — until the president said, ‘They’re all politicians! Where’s Einstein? Or Jonas Salk? Is the curriculum too focused on political science or politics?’

“Believe it or not, there will be a faculty meeting coming up where this is the entire agenda,” Light said. “The take-away is straightforward: A little bit of simple data can affect curriculum and requirement policies.

“Whether any or all of these ideas would work at Pitt is not for me to say,,” Light concluded. “But feel free to take these ideas and run with them.”

The annual Pitt Symposium on Undergraduate Advising is co-sponsored by the College of Business Administration, the College of General Studies, the Arts and Sciences (A&S) Advising Center, the Division of Student Affairs, the School of Education, the School of Engineering, the Office of the Associate Dean for Undergraduate Studies in A&S, the University Honors College and the regional campuses.

—Peter Hart

Filed under: Feature,Volume 38 Issue 5

Leave a Reply