Sarah L. Eddy and Kelly A. Hogan (2014) recently published a paper “Getting Under the Hood: How and for Whom Does Increasing Course Structure Work?”, a nice example of the next wave of discipline-based educational research (DBER) that goes beyond asking “Does active learning work?” to explore details of how active learning interventions actually work, and differential impacts on sub-populations of students. Here, Eddy and Hogan describe their results of a study based on the work led by Scott Freeman at the University of Washington (see Freeman et al. 2011, Haak et al. 2011).

What were they studying?

The earlier studies indicated that increasing course structure reduced failure rates in an introductory majors biology course, with disadvantaged students (including under-represented minorities and first generation students) reducing the typical achievement gap. What is meant by “increased course structure”? Freeman et al. (2011) describe “high structure” courses as using lots of active learning exercises (e.g., minute papers, case studies, in-class group exercises), and formative assessments (online practise questions, clicker questions), while “low structure” courses use traditional lecturing and a few high stakes assessment (midterms + final exam). Table 1 from Eddy and Hogan (2014) breaks this down further into high/moderate/low structure.

Eddy and Hogan carried out their own study based on these findings in a new context – different instructors, different institution, different student body, and in a non-majors biology course. They also carried out some interesting analyses disaggregating ethnic/racial groups (rather than considering all under-represented minorities [URM] as a single group), and looking at factors that may have contributed to improved student performance.

What did they find?

Increased course structure was associated with general improvement in exam performance. (In this work, the increased structure was in the form of ungraded guided-reading questions, graded preparatory homework questions, classroom-response questions for bonus points, resulting in a “moderate structure” course.) Overall, scores improved by 3.2%. The increased structure was particularly beneficial for black students (with an additional 3.1% increase to predicted exam grade) and first generation students (additional 2.5%). As in the 2011 studies, failure rates dropped in the higher-structure course. (“Failure” here is defined as a C- or lower.)

The authors found evidence of some specific factors in the more highly structured courses that may have contributed to the improvements in student performance. In surveys, students reported increased time spent on coursework, likely because of the accountability associated with structured elements of the course. An improved sense of community was also reported (although other surveyed aspects of classroom culture did not seem to vary significantly). Sense of community is a factor that relates to STEM retention issues, particularly for students of colour and women.

Students were also surveyed regarding their perceptions of the value of the course, and overall there was no evidence that perceptions of the course value significantly differed by structure. However, black students did report different perceptions of the more highly structured course, finding more value in the homework, less memorization/more higher-order skills, and a perception that skills learned in this course would be useful to them.

The authors highlight the diversity of university students, and the need to consider whether particular interventions will benefit various populations. (This is not wholly new – e.g., see Nelson 1996, Treisman 1992 – but it is true that many studies have lumped URM into a single bucket.) The need for accountability in changing student behaviours is also stressed – the need to associate activities with grades/credit. They also felt that student surveys were valuable in identifying important/relevant student behaviours associated with performance.

Some of my thoughts/questions …

First of all, we need more carefully designed DBER studies like these. There was clearly careful study design, and variables were considered and controlled for (e.g., cognitive levels of questions used, controlling for previous academic achievement). I was impressed that there were actual measurements of what time was spent on in class by observing classroom videos (rather than instructor self-reporting). The better the evidence in the literature, the more persuasive we can be in encouraging better practises in higher education. Having listed examples of practical methods that could be incorporated in many courses to increase structure is very helpful.

Eddy and Hogan suggest that increasing course structure may be a more cost-effective and sustainable strategy than some other initiatives, like supplemental instruction (also discussed in the Freeman et al & Haak et al 2011 papers). Eddy and Hogan state “one instructor in one course, without a large influx of money, can make a difference for students as large in magnitude as some supplemental instruction programs”.  This seems possible, and good reason to encourage educators to use evidence-based methods that would increase course structure. Still, I must ask: How much time and effort is required by the “one instructor” to make these changes, and sustain them? I may be unusually inefficient, but when I’ve added elements increasing course structure, this always takes some additional time and effort (especially compared to my early, lecture-heavy teaching years). I don’t begrudge this investment (usually) – there is evidence such methods help student learning, and often make the classroom a more dynamic and interesting place for me, too. Nevertheless, this increase in workload (e.g., making up online quiz questions, developing in-class activities, marking worksheets) is not insignificant, and generally not externally recognized/valued, at least in my experience.

In discussing student time allocation and accountability, Eddy and Hogan highlight that preparation for class can take various forms … but to be effective, they must be graded. This makes sense – we all prioritize our tasks based on the apparent (and, often short term) rewards, and accountability shifts priorities – but grading adds a clear burden of work on the instructor (and/or TAs, if you’ve got them). Even if you use machine-scored questions (e.g., in an online quiz, or for scantrons), coming up with good score-able questions does require considerable preparation. If you do this each year in the same course, you’ll likely need to update your questions each year, too. As educators, we should be striving to use the best methods to help improve teaching … within reason. I worry that there is increasing pressure on the educator to perform better to make up for the fact that we are facing increasing class sizes and often reduced resources/support. One instructor can make a difference. However, there are limits to what individual instructors can do within the larger educational system.

What makes a student value a course? Unexpectedly, the authors saw an overall decline in perception of course value to students! It’s been a common discussion point (among my colleagues, at education conferences and oCUBE unconferences) about factors that are associated with effective learning vs. what students may perceive as helping them learn. With increased course structure, I think that there is the possibility of push back from students, who may not recognize the value of the work expected of them (compared what is expected in lower-structure courses). This perhaps relates to the authors’ findings regarding student survey responses of their perceptions of course value.  Although Eddy and Hogan report a clear benefit relating to the increased course structure in terms of student exam performance,  this does not seem to have been perceived by most of the students in their study (or, at least, related to the student reports of perceived value). As many university and government initiatives currently focus on improving the “student experience”, measurements of success often take the form of student opinion surveys. What happens when student perceptions aren’t aligned with their own successes?

Where do we go from here?

The study’s authors indicate more work needs to be done, and highlight the value of disaggregating the “under-represented minority” group when conducting studies on educational interventions, given the diversity of individuals within that category. Many universities in Canada have highly diverse populations. As in the United States, the demographics will vary between our institutions. It would be interesting to see similar studies undertaken at large schools like York University, The University of Toronto, and the University of British Columbia, as a starting point, to get some idea of how more structured courses might impact Canadian students, including first generation students, and those from various ethnic/racial backgrounds. Unfortunately, there seem to be fewer funding opportunities for studying higher education in Canada than in the United States, particularly for larger projects. I am hopeful that a group like the Carl Wieman Science Education Initiative (CWSEI) at UBC might undertake something like this. (Actually, they have carried out a number of initiatives involving active learning, and I wouldn’t be surprised if they’ve collected some useful information regarding effects of specific interventions on student sub-populations.)   The studies mentioned here investigated effects of increased structure in introductory courses. Would increased course structure in upper level courses be as beneficial? Presumably, upper-year students have developed some self-regulatory skills-perhaps other interventions (to encourage more advanced learning approaches) might be more valuable for them. It would be wonderful to have clear evidence of which strategies would be most effective to use when teaching for one’s discipline, type of course, and student body.

A question that almost always comes to my mind when I read these types of studies is how we, as educators, can improve our courses, without much in the way of additional resources (if any). Where are my efforts going to have the most impact? I’m already using structured elements in my classes (including online pre-class quizzes, a classroom response system, and occasional worksheets and online assignments) – can I add more? (I’m thinking about using PeerWise next term.) Even some of the technologies that can help facilitate these activities have a learning curve for a new user, something to factor in the time needed to implement them. Are there better ways to increase accountability for student behaviours that help with learning … without having to remove something else that takes instructor time?

I’d be interested in hearing your suggestions (along with any other related comments/questions you have) on this!

References:

Eddy, S. L., & Hogan, K. A. (2014). Getting under the hood: how and for whom does increasing course structure work?. CBE-Life Sciences Education, 13(3), 453-468.

Freeman, S., Haak, D., & Wenderoth, M. P. (2011). Increased course structure improves performance in introductory biology. CBE-Life Sciences Education, 10(2), 175-186.

Haak, D. C., HilleRisLambers, J., Pitre, E., & Freeman, S. (2011). Increased structure and active learning reduce the achievement gap in introductory biology. Science, 332(6034), 1213-1216.

Nelson, C. E. (1996). Student diversity requires different approaches to college teaching, even in math and science. American Behavioural Scientist, 40, 165-175.

Treisman, U.  (1992). Studying students studying calculus: A look at the lives of minority mathematicians. A Mary P. Dolciani Lecture. College Mathematics Journal, 23, 362-372.