National Survey of Student Engagement (NSSE)

Earlier this week the USA Today published their first report on NSSE. The fact that institutions were reluctant to share their NSSE results was noted in Insidehighered.com but this was the first time that NSSE results have been broadly published in the USA. At McMaster, like every other university in the Province, we are preparing to do NSSE in early 2008. As I’ve noted in previous posts, NSSE is by far the best tool for assessing the quality of universities. I say this notwithstanding the fact that Macleans, Canada’s most famous university ranking magazine, issued their rankings today.
What is NSSE? A good place to learn is the NSSE home page and the best document to explain the theory behind the NSSE is entitled “Conceptual Framework and Overview of Psychometric Properties” by George Kuh. NSSE is not a student satisfaction survey (like the Canadian University Survey Consortium or the yearly Globe & Mail “University Report Card“. Nor is NSSE a ranking based on input measures like the one published today by Macleans. NSSE is meant to measure “student engagement”. The implication is that the more students are engaged in the learning process both in the classroom or lab and outside the classroom; the more they will learn and develop. There is plenty of theory available to back up this claim as noted in the references to Kuh’s paper. The paper also validates the use of self-reported data from students. According to Kuh: “…a good deal of evidence shows that students are accurate, credible reporters of their activities and how much they have benefitted from their college experience”. The Canadian version of the NSSE instrument was only available in an on-line version while the U.S. version is available in hard-copy.
The questions posed in the NSSE survey are meant to measure how students are engaged at university and are, in my opinion, based on three types of research. The first type is exemplified by the paper: “Seven Principles for Good Practice in Undergraduate Education” by Chickering and Gamson (1987). These seven principles for good practice (according to Chickering and Gamson) are:

1. encourages contact between students and faculty
2. develops reciprocity and cooperation among students
3. encourages active learning,
4. gives prompt feedback
5. emphasizes time on task
6. communicates high expectations
7. respects diverse talents and ways of learning.

The questions under #1. in the NSSE survey relate to these principles (for example question 1q. asks about prompt feedback); question 3 relates to high expectations, questions 4,5 relate to time on task, and so on. Other questions ask about faculty student interactions, student co-operation and active learning. A good example of communicating high expectations is question 1r (“have you ever worked harder than you thought you could to meet an instructor’s standards or expectations?)
An example of the second type of research used by NSSE is Bloom’s (1984) taxonomy classification. This single page site is re-printed here.

1. Knowledge: arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce state.
2. Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate,
3. Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write.
4. Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.
5. Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write.
6. Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate.

Basically, at level 1 of Bloom’s Taxonomy (the lowest level – knowledge) we are talking about simple delivery of facts. For example in a calculus course, we learn how to take a derivative. At level 2 we learn that a derivative is a rate of change. That is, we understand the meaning. Then at level 3 we use this understanding to solve a practical problem in an application – like how long it takes to drain a container of water for example. At the higher levels we ask students to do analysis and synthesis – in engineering courses for example this means design – and finally at level VI students need to reflect on whether their solution or design makes sense. In a first year calculus course we expect the first 3 levels. In a fourth year capstone design course we expect mainly levels 4 to 6. Question 2 of the NSSE survey deals almost entirely with the various levels of this taxonomy and one of the NSSE benchmarks (called Level of Academic Challenge (LAC) and discussed later) is heavily weighted by the responses to this question. A companion survey to NSSE is FSSE (Faculty Survey of Student Engagement) where faculty members are asked a similar suite of questions. It would be interesting to compare the perceptions of students with those of their professors – especially as they relate to Bloom’s Taxonomy.

An example of the 3^rd type of research is described in Alexander Astin’s (1996) book – “What matters in College- Four critical years revisited”. A paper by Felder (1993) from the Chemical engineering literature discusses Astin’s book and the following quotes from Felder’s paper (quoting key parts of Astin’s book) apply:

“The quality of the college experience is strongly affected by student-faculty interactions. The frequency with which students talk with professors outside class, work with them on research projects, assist them in teaching, and visit their homes, correlates with student grade-point average, degree attainment, enrollment in graduate or professional school, every self-reported area of intellectual and personal growth, satisfaction with quality of instruction, and likelihood of choosing a career in college teaching [pp.383-384].”

“Astin concludes, however, that as important as the student-faculty relationship may be, “…the student’s peer group is the single most potent source of influence on growth and development during the undergraduate years.”[p. 398] Frequency of student-student interactions (including discussing course content with other students, working on group projects, tutoring other students, and participating in intramural sports) correlates with improvement in GPA, graduating with honors, analytical and problem-solving skills, leadership ability, public speaking skills, interpersonal skills, preparation for graduate and professional school, and general knowledge, and correlates negatively with feeling depressed [p. 385].”

Basically these interactions are extremely important, as are group work and co-operative learning. A number of the questions in #1 & #7 of the NSSE survey are meant to tease out these attributes of Student Faculty Interaction (SFI) and form the basis of another NSSE benchmark

The two summative questions of the survey are: #13 “How would you evaluate your entire educational experience at McMaster”; and, #14 “If you could start over again, would you go to the same institution you are now attending”. Responses to questions #13 and #14 are published by each Ontario University in its common data set (CUDO). McMaster’s CUDO site also has a link to its other NSSE data .

Benchmarks

In addition to mean and frequency reports for each question, a second way that NSSE uses to present the survey results is in a Benchmark Comparisons Report. Benchmarks (BMs) represent a composite or cluster of responses to several questions from the survey and NSSE uses five different ones for each level. The 5 BMs are: Level of Academic Challenge (LAC), Active and Collaborative Learning (ACL), Student-Faculty Interaction (SFI), Enriching Educational Experiences (EEE) and, Supportive Campus Environment (EEE).

The responses to the following question numbers from the survey are used to comprise the benchmarks.

LAC: Challenging intellectual and creative work is central to student learning. Students are asked, for example, how much time they spent preparing for class, number of written papers or reports they write, and whether the coursework emphasizes things like synthesizing ideas or applying theories to practical problems.

Survey questions: Q1(r),Q2( b, c, d, e),Q3(a, c, d, e), Q9(a), Q10(a)

ACL: Students learn more when they are intensely involved in their education. Students are asked, for example, how often they participated in class discussions, made a class presentation, worked with classmates outside of class to prepare assignments, or discussed ideas from readings outside of class.

Survey questions: Q1(a, b, g, h, j, k, t)

SFI: Contact with professors offers students an opportunity to see how experts think about and solve practical problems. Students are asked, for example, whether they have worked with faculty members on activities outside of coursework, received prompt written and oral feedback, or worked with a faculty member on a research project.

Survey questions: Q1(n, o, p, q, s), Q7(d)

EEE: Activities that complement the classroom experience can make learning more meaningful. Students are asked, for example, whether they have had serious conversations with students who have different beliefs or values, or with students of a different race/ethnicity. They’re also asked whether they have studied abroad and participated in student clubs, learning communities, internships and culminating senior experiences.

Survey questions: Q1(l, u, v), Q7(a ,b, e, f, g, h),Q9(d), Q10(c)

SCE: Students perform better when their college is committed to their success and cultivates positive social relationships among different groups of people. Students are asked, for example, whether the campus provides “the support you need” to succeed academically and thrive socially and to assess, for example, the quality of their relationships with other students, faculty and the administration.

Survey questions: Q8(a, b, c),Q10(b, d, e)

Basically NSSE results are not to be used for ranking purposes. McMaster’s use of the results is typical of other Ontario Universities – that is, the Benchmark Report was posted on the University’s web page. Last Spring Macleans collected Benchmark scores from several Canadian universities.

Results from other significant questions

The questions asked in Q#11 of the survey are related to a student’s educational development and personal growth. However the responses to Q#11 do not fit within any NSSE benchmark. Nonetheless, to me at least, they are amongst the most important in the survey as they demonstrate (self-reported) growth in student development as well as the “soft skills” (writing skills, speaking skills, critical thinking skills, etc.) so highly valued by the employers of our students. The term soft skills is politically charged in some faculties – most of us would call them “vital skills” that we expect all of our graduates (regardless of discipline) to demonstrate.

Student Satisfaction Questions

Although the majority of the NSSE questions deal with engagement there are three that measure student satisfaction and two of them are most widely used for comparison purposes. They are: Q#13. How would you evaluate your entire educational experience at this institution? and, Q#14. If you could start over again, would you go to the same institution you are now attending? The results below have been extracted from the NSSE 2006 Mean and Frequencies comparisons found on McMaster’s website.

[ Click here ] to view McMaster’s NSSE Results (a new window will open).

These results were taken from the spreadsheet NSSE06 Mean and Frequency Report (McMaster).xls which is posted on the McMaster Website . So, for example, 266 first year student rated their Experience at McMaster to be excellent. In the data above: “Ontario” refers to the COU consortium with responses from McMaster removed. The Carnegie Peers (that McMaster chose are: Auburn University, Case Western Reserve, Emory University, Georgetown University, Loyola University College, Saint Louis University, University of Arizona, University of Tennessee, University of Miami, University of Michigan – Ann Arbor, University of Missouri – Columbia, University of Pittsburgh, University of Wisconsin – Madison, Virginia Commonwealth University, Washington State University, Wayne State University) and NSSE Refers to the entire NSSE 2006 sample.

[ Click here ] for NSSE 2006 Mean Comparisons (a new window will open)

 

I’ll have more to say about NSSE and the other ranking and student satisfaction surveys in future posts.

 

References

Astin, A.A., “What matters in College: Four Critical Years Revisited”, Jossey-Bass, (1993).

Bloom, B. S. Taxonomy of educational objectives, Allyn and Bacon, Boston, MA. (1984)

Burgan, M., “In Defense of Lecturing”, Change November/December (2006)

Chickering, A.W. and Gamson, Z.F., “Seven Principles for Good Practice in Undergraduate Education”, The Wingspread Journal, 9, 2, (1987)

Felder, R. “What Matters in College.” Chem. Engr. Education, 27(4), 194-195 (Fall 1993).

Pascarella, E.T. and Terenzini, P.T., How College Affects Students, Jossey-Bass, (1991)