A Cold Shot to Responsible Drinking

I was going to write about the Macleans magazine University Rankings this week but events late last week caused a change in plans. Hopefully I’ll have something to say about Macleans later this week or early next week. Last Wednesday evening I received the press release: “Agency announces launch of Molson Canadian Cold Shots Campus Challenge”. According to the release: “The interactive campaign leverages the social networking site, Facebook, in order to increase brand awareness and drive product trial and retrial” (emphasis mine). Apparently there is a “Molson Canadian Nation” group on Facebook and this was judged to be the key tool to reach the crucial 19-24 demographic. “By utilizing Facebook’s inherent viral capabilities, the Cold Shots Campus Challenge will be easily spread throughout the different school networks in order to drive greater user participation.” (This is all ad agency lingo).

Again, quoting directly from the press release: “The Cold Shots Campus Challenge involves a user generated content component that pits academic institutions against each other for the title of #1 Party School in Canada. Students are prompted to upload pictures showing how they get the party started with their friends before a night out. The images are posted within virtual dorm rooms on Facebook – each dorm room associated with a different university or college. (Note that essentially no one living in a dorm at McMaster University is old enough to legally drink a Molson Cold Shot!)

My colleague at Queen’s University, Jason Laker had already been contacted by the Globe and Mail asking for his comments on the promotion and he sent members of the Ontario Committee on Student Affairs (OCSA – the committee of all senior student affairs officers (SSAO) in the province) a press release announcing the promotion. By Thursday at noon I was contacted by the Globe and Mail and repeated much of what Jason had said, that is, that we were appalled by the campaign since it is anathema to all that we do related to responsible drinking not to mention the very inappropriate use of Facebook. Essentially I thought that Molson was being irresponsible. The reporter’s story appeared on page B7 of the Globe on Friday morning.

You will note that in the article I said that we would be sending a letter to Molson. I was at a meeting in Toronto on Friday when I received an e-mail from a VP at Molson suggesting that I send such a letter to him. I had previously written the letter and sent an e-mail back to Mac asking that it be put on letterhead for my signature. Jason also received an e-mail from the same VP and sent him a scathing reply suggesting: “I encourage you to cancel this contest and marketing campaign while the public discussions are in their early stages (though it seems to be escalating very quickly), to express regret for it, and to offer to meet with professional organizations and administrators to identify ways for Molson to make a real difference in reducing the epidemic of alcohol abuse and its consequences. I would be very happy to engage with you in this regard.” Meanwhile other SSAOs and student government leaders were writing letters and engaging groups like studentlifeeducation.com and Bacchus – the alcohol education group. It seems that all of this agitating had an impact as the contest was cancelled late Friday afternoon – before I had even arrived home from Toronto.

I was contacted by the Bloomberg organization Monday morning for comment and a small part of my interview appears here . In both of the interviews I have tried to discuss with the writers why people in my position are so offended by the Cold Shots/Facebook promotion. The most cogent arguments are those of my colleague at the University of Guelph, Brenda Whiteside. Brenda is the Chair of OCSA and wrote the following in her letter from OCSA to Molson:

“As professionals dealing with the transition of young students to university, we are continuously combating the stereotype of universities being places to party – a stereotype that derives from movies such as Animal House. Many of our first year students enter university never having been away from home and are experimenting with many new situations. One of the risks of a “party university” culture is the increased potential for over consumption of alcohol by students with no previous experience. Each year we send a small but concerning number of students to hospital with alcohol poisoning. Some of us have had to deal with a student death due to over consumption. In addition, we continually struggle with the by-products of over consumption – vandalism, assault, sexual assault, and academic failure. Like Molson we devote endless energy, resources and programming towards responsible drinking campaigns. We applaud Molson’s leadership in this area as indicated on your webpage “with responsible drinking program, Molson, Canada’s favourite brewer, once again leads the way, a champion in celebrating consumers making responsible choices”. Imagine our dismay when we learned of this campaign – a competition for a party environment in residence. This campaign not only dismantles university efforts to create a culture of academics in residences, it also helps to establish an environment that could encourage irresponsible drinking.

Our second concern with this campaign is the use of Facebook. Facebook is an extraordinarily powerful social networking tool and we can understand Molson’s interest in leveraging this tool. However, Facebook has the potential to be dangerous for students and all Ontario universities are engaged in educational programs that warn students of the dangers of Facebook. Of particular concern is students posting pictures of themselves in behaviour that could impact negatively on future careers or opportunities. There are numerous examples of students who have regretted the posting of such pictures. Thus, please understand again our dismay with a program that encourages students to potentially place such damaging photos on the internet.

We recognize that Molson’s has had a long and proud history of commitment to responsible drinking. We have been impressed with Molson’s launch of the Plan Ahead Don’t Drink and Drive program and its partnership with the Student Life Education Company. This present campaign runs counter to Molson’s commitment to responsible drinking. We hope that your company will be more thoughtful about possible risks of advertising campaigns prior to future launches. We thank you for your thoughtful consideration of our concerns and are always available to consult on issues that relate to university students.”

I am very proud of the efforts of Brenda, Jason and several others who took the time to nip this promotion in the bud. As Brenda mentioned in her letter, we deal with the consequences of over-consumption every week. As I mentioned in an earlier post on Web 2.0, I had to suspend a student for the inappropriate use of Facebook. Last year we had to fire some Residence CAs (Dons) who had posted pictures of themselves drinking while they were on duty. These events caused us such concern that we initiated a poster campaign warning students against potential misuse of Facebook.

It’s clear to me that Molson was misguided in its attempt at using the social networking site to market its products. “We were looking for a way to reach our target demographic in the most efficient manner,” said Sarah Eby, Brand Manager with Molson Canadian.

“Creating a campaign on Facebook allows us to leverage an environment that already exists, and one that really promotes social interaction.” A quote in the press release from their ad agency: “Utilizing Facebook for this program made strategic sense,” said Heather Clark, Creative Strategy Director with henderson bas. “This is where they hang out and it’s a great opportunity to engage them in a dialogue. If you do it right, you can find yourself with a whole nation of brand ambassadors.” suggests that they also didn’t know that students resent corporations blatant advertising on what they view as their own site. Response to the stories in the Globe was most critical of this aspect of the claim.

It is not clear whether or not Molson has learned their lesson from this fiasco. A quote in the second Globe story from the VP mentions above: “”The whole realm of social media – there’s lots to learn,” he said.”It’s really a new area. We’re probably groundbreaking and leading in a lot of things we’ve been doing.” suggests that they haven’t.

Ranking Universities

The ranking season is upon us. What started in the U.S. with the US News and World Report moved to Canada 16 years ago when Macleans did their first ratings. Since then, in Canada, we have had new satisfaction surveys done by the Canadian University Consortium (CUSC) and more recently The Globe and mail as part of their University Report Card.A frequent commentator on university rankings is Alex Usher, Vice President of the Educational Policy Institute (EPI). In a recent article, Usher taught me a new word hermeneutics -which I find to be defined as “the science of interpretation, especially of the Scriptures”. He has certainly been religious in his coverage of the issue. To quote from the hermeneutics article: “Most rankings are fundamentally measuring – either directly or indirectly – institutional funding (Maclean’s is notably heavy on these kinds of measures) and research output (e.g. Shanghai Jiao Tong and other Asian rankings systems).  Many also measure something called “prestige” or “reputation” (the Times Higher Education Supplement world rankings is particularly heavy on this measure, but it is US News and World Report that gets a lot of the press on this one).  Some people say this highly unfair because it is “too subjective”.  But as recent research by Ross Williams of the Melbourne Institute and Gero Federkeil of the German Centre for Higher Education has shown, “reputation among academics”, properly measured,  is actually an excellent proxy for research output because of the high degree of correlation between reputation and bibliometric measures of scientific output.  So in fact, when rankings measure prestige, they are actually indirectly measuring research output”.

Usher and the EPI have been involved in the Canadian rankings scene as well. Last year, together with the Strategic Counsel, they helped the Globe and Mail develop their “University Navigator” Usher’s credibility took a hit when he published the paper: Can our schools become word class?” in the 2006 University Report Card issue. His definition of word class universities: “World-class, then, means large, expensive, research-intensive institutions. That rules out the majority of Canada’s universities: There are really only about 20 institutions that have the required size to even be considered as having world-class status. This does not mean there are no world-class academics in other institutions, just that they do not collectively have the resources to make their institution world-class.” Is simply wrong. Most of the universities in the top 25 list in the article (taken from the Shanghai Jiao Tong ratings )are small. Indeed, Caltech which is listed as #6 in 2006 and 2007 has fewer than 1000 undergrads and about that many grads. For some peculiar reason, the table in his paper also did not list McMaster University which, in addition to the three Canadian universities listed, is also ranked in the Top 100 in the world (#87).

Usher’s comment: “It’s also a matter of tuition fees, since large U.S. research institutions typically charge higher fees than their Canadian counterparts. Fortunately, most of them also have extremely generous need-based student aid packages for undergraduates from lower-income backgrounds. “  demonstrates a lack of knowledge about how tuition fees enter the equation at research intensive universities. Graduate students typically don’t pay tuition fees. The fees are waived by the student’ supervisor must raise the money for the fees from a grant. The grant would also contain an allowance for overhead based on the value of the fees (typically 70%). So a student whose fees were nominally $30,000 would require the advisor to have approximately $50,000 in a grant from a government agency, foundation or company.

Nonetheless, I do believe that Alex Usher is the most knowledgeable commentator on university rankings in Canada. In a recent report Usher blasts the Times Higher Education Supplement rankings because of the unreasonably high rankings that they give to some Canadian Universities. In the following I will discuss the recent ratings reported in the Globe and Mail. Next week I’ll report on Macleans.

Globe and Mail – University Report Card (URC)

The Globe and Mail’s URC has evolved since its original appearance in 2002. The survey used to be conducted by having students go to the website studentawards.com where they were deluged with advertising before going to complete a student satisfaction survey.  The survey was not properly administered and there were few checks on the administration of the survey. It came under considerable criticism in 2003 when students were asked to rate “Post-Grad by Faculty” and rated the University of Waterloo #9 for Law. Waterloo does not have a Faculty of Law.

Over the years  the survey has improved. First the Strategic Counsel was engaged and more recently the EPI as previously noted. For the 2006 survey, as a pilot, some universities were asked to participate. McMaster participated in the 2007 survey. By participation what is mean that e-mails were sent to students inviting them to go to a website to complete the survey. I’m not sure how many invitations were sent out but 348 McMaster students replied. In total 43,222 students from across Canada participated representing 53 universities. Of the 53 universities 42 were active participants, that is, in a manner similar to McMaster they contacted their own students.

The results of the survey are published in a number of ways. A hard copy magazine was delivered as an insert in the Globe on November 2^nd . More details are published in the hard to find Education section of the Globe’s website. The magazine contains ratings in approximately 20 categories – mostly the overall ones. The website has ratings for 78 questions. As the administrator responsible for McMaster’s participation I was sent a complete set of responses for McMaster students. In fact there were over 200 questions asked. Many of the questions are similar to those of the NSSE survey discussed last week. Of course the responses to most of these questions were not published and it remains to be seen how the Globe will use the data.

Over the years McMaster has fared pretty well in the URC. This year for the first time we were included as one of the 15 “Large” schools (those with > 22,000 students). The key overall categories are “Quality of Education” and “Most Satisfied Students” (corresponding to NSSE questions #14 and #15). In Quality McMaster received an A- which was the second highest grade in our category. Only Western received a higher grade (A). The grades are tightly bunched however with the lowest being B. In Satisfied Students McMaster received a B+ which was the fourth highest grade given. Western again received an A while Alberta and Waterloo received an A-. McMaster published a brief article on the URC but otherwise does not do too much with the ratings.

Our results in individual categories were both good and surprising. The Bookstore and Campus Health continue to be very highly rated (amongst the best in the country). Our athletic programs rating increased from B+ to A- and recreation facilities from B (nearly last) to A- (behind only 2 Quebec schools). These ratings reflect the opening last year of the $30 million David Braley Athletic Centre. Unfortunately our rating for variety of extracurricular activities, which had been steady at A, dropped to B+ for no apparent reason. In addition our rating for school spirit which was A+ in 2004 was A- in 2005, A- in 2006 and now B in 2007. Over that time the university has grown considerably and this rating (and others) probably reflects that increase.

Again, because of our rapid growth, services have become strained. In particular students have no place (few places) for quiet study. Our rating here dropped from B (last in the medium category) last year to C+ (last in the large school category) this year. The overall library rating has dropped from a consistent A- to a B this year. This was the lowest rating given in our category.

My feeling on these sorts of ratings is two fold. Students will often parrot what they hear. The University of Western Ontario tells students: “Western provides the best student experience among Canada’s leading research-intensive universities” prominently on their home web page. They have made the student experience a key in their rhetoric and recruiting. McMaster was Canada’s research university of the year a few years ago and we proudly trumpeted this achievement. Naturally McMaster students rate McMaster’s reputation for research highly. Western’s students likewise give high grades to all things student.

Managing expectations is also difficult. Over the past four years our grade for food services has dropped from a B+ to a C+. We are still rated in the top few in our category but it is impossible to meet the expectations of our customers (good food, plentiful, fresh, cheap and with infinite variety). Some universities do a better job of trumpeting their successes but our food services win awards every year  – including major North American ones. The story is similar for financial aid. A B is a very good rating. Financial aid offices could never give away enough money. On the other hand a student’s perception of safety is always rated high. Most schools get an A and we are no exception.

Next post will deal with the recent Macleans ratings.

National Survey of Student Engagement (NSSE)

Earlier this week the USA Today published their first report on NSSE. The fact that institutions were reluctant to share their NSSE results was noted in Insidehighered.com but this was the first time that NSSE results have been broadly published in the USA. At McMaster, like every other university in the Province, we are preparing to do NSSE in early 2008. As I’ve noted in previous posts, NSSE is by far the best tool for assessing the quality of universities. I say this notwithstanding the fact that Macleans, Canada’s most famous university ranking magazine, issued their rankings today.
What is NSSE? A good place to learn is the NSSE home page and the best document to explain the theory behind the NSSE is entitled “Conceptual Framework and Overview of Psychometric Properties” by George Kuh. NSSE is not a student satisfaction survey (like the Canadian University Survey Consortium or the yearly Globe & Mail “University Report Card“. Nor is NSSE a ranking based on input measures like the one published today by Macleans. NSSE is meant to measure “student engagement”. The implication is that the more students are engaged in the learning process both in the classroom or lab and outside the classroom; the more they will learn and develop. There is plenty of theory available to back up this claim as noted in the references to Kuh’s paper. The paper also validates the use of self-reported data from students. According to Kuh: “…a good deal of evidence shows that students are accurate, credible reporters of their activities and how much they have benefitted from their college experience”. The Canadian version of the NSSE instrument was only available in an on-line version while the U.S. version is available in hard-copy.
The questions posed in the NSSE survey are meant to measure how students are engaged at university and are, in my opinion, based on three types of research. The first type is exemplified by the paper: “Seven Principles for Good Practice in Undergraduate Education” by Chickering and Gamson (1987). These seven principles for good practice (according to Chickering and Gamson) are:

1. encourages contact between students and faculty
2. develops reciprocity and cooperation among students
3. encourages active learning,
4. gives prompt feedback
5. emphasizes time on task
6. communicates high expectations
7. respects diverse talents and ways of learning.

The questions under #1. in the NSSE survey relate to these principles (for example question 1q. asks about prompt feedback); question 3 relates to high expectations, questions 4,5 relate to time on task, and so on. Other questions ask about faculty student interactions, student co-operation and active learning. A good example of communicating high expectations is question 1r (“have you ever worked harder than you thought you could to meet an instructor’s standards or expectations?)
An example of the second type of research used by NSSE is Bloom’s (1984) taxonomy classification. This single page site is re-printed here.

1. Knowledge: arrange, define, duplicate, label, list, memorize, name, order, recognize, relate, recall, repeat, reproduce state.
2. Comprehension: classify, describe, discuss, explain, express, identify, indicate, locate, recognize, report, restate, review, select, translate,
3. Application: apply, choose, demonstrate, dramatize, employ, illustrate, interpret, operate, practice, schedule, sketch, solve, use, write.
4. Analysis: analyze, appraise, calculate, categorize, compare, contrast, criticize, differentiate, discriminate, distinguish, examine, experiment, question, test.
5. Synthesis: arrange, assemble, collect, compose, construct, create, design, develop, formulate, manage, organize, plan, prepare, propose, set up, write.
6. Evaluation: appraise, argue, assess, attach, choose compare, defend estimate, judge, predict, rate, core, select, support, value, evaluate.

Basically, at level 1 of Bloom’s Taxonomy (the lowest level – knowledge) we are talking about simple delivery of facts. For example in a calculus course, we learn how to take a derivative. At level 2 we learn that a derivative is a rate of change. That is, we understand the meaning. Then at level 3 we use this understanding to solve a practical problem in an application – like how long it takes to drain a container of water for example. At the higher levels we ask students to do analysis and synthesis – in engineering courses for example this means design – and finally at level VI students need to reflect on whether their solution or design makes sense. In a first year calculus course we expect the first 3 levels. In a fourth year capstone design course we expect mainly levels 4 to 6. Question 2 of the NSSE survey deals almost entirely with the various levels of this taxonomy and one of the NSSE benchmarks (called Level of Academic Challenge (LAC) and discussed later) is heavily weighted by the responses to this question. A companion survey to NSSE is FSSE (Faculty Survey of Student Engagement) where faculty members are asked a similar suite of questions. It would be interesting to compare the perceptions of students with those of their professors – especially as they relate to Bloom’s Taxonomy.

An example of the 3^rd type of research is described in Alexander Astin’s (1996) book – “What matters in College- Four critical years revisited”. A paper by Felder (1993) from the Chemical engineering literature discusses Astin’s book and the following quotes from Felder’s paper (quoting key parts of Astin’s book) apply:

“The quality of the college experience is strongly affected by student-faculty interactions. The frequency with which students talk with professors outside class, work with them on research projects, assist them in teaching, and visit their homes, correlates with student grade-point average, degree attainment, enrollment in graduate or professional school, every self-reported area of intellectual and personal growth, satisfaction with quality of instruction, and likelihood of choosing a career in college teaching [pp.383-384].”

“Astin concludes, however, that as important as the student-faculty relationship may be, “…the student’s peer group is the single most potent source of influence on growth and development during the undergraduate years.”[p. 398] Frequency of student-student interactions (including discussing course content with other students, working on group projects, tutoring other students, and participating in intramural sports) correlates with improvement in GPA, graduating with honors, analytical and problem-solving skills, leadership ability, public speaking skills, interpersonal skills, preparation for graduate and professional school, and general knowledge, and correlates negatively with feeling depressed [p. 385].”

Basically these interactions are extremely important, as are group work and co-operative learning. A number of the questions in #1 & #7 of the NSSE survey are meant to tease out these attributes of Student Faculty Interaction (SFI) and form the basis of another NSSE benchmark

The two summative questions of the survey are: #13 “How would you evaluate your entire educational experience at McMaster”; and, #14 “If you could start over again, would you go to the same institution you are now attending”. Responses to questions #13 and #14 are published by each Ontario University in its common data set (CUDO). McMaster’s CUDO site also has a link to its other NSSE data .

Benchmarks

In addition to mean and frequency reports for each question, a second way that NSSE uses to present the survey results is in a Benchmark Comparisons Report. Benchmarks (BMs) represent a composite or cluster of responses to several questions from the survey and NSSE uses five different ones for each level. The 5 BMs are: Level of Academic Challenge (LAC), Active and Collaborative Learning (ACL), Student-Faculty Interaction (SFI), Enriching Educational Experiences (EEE) and, Supportive Campus Environment (EEE).

The responses to the following question numbers from the survey are used to comprise the benchmarks.

LAC: Challenging intellectual and creative work is central to student learning. Students are asked, for example, how much time they spent preparing for class, number of written papers or reports they write, and whether the coursework emphasizes things like synthesizing ideas or applying theories to practical problems.

Survey questions: Q1(r),Q2( b, c, d, e),Q3(a, c, d, e), Q9(a), Q10(a)

ACL: Students learn more when they are intensely involved in their education. Students are asked, for example, how often they participated in class discussions, made a class presentation, worked with classmates outside of class to prepare assignments, or discussed ideas from readings outside of class.

Survey questions: Q1(a, b, g, h, j, k, t)

SFI: Contact with professors offers students an opportunity to see how experts think about and solve practical problems. Students are asked, for example, whether they have worked with faculty members on activities outside of coursework, received prompt written and oral feedback, or worked with a faculty member on a research project.

Survey questions: Q1(n, o, p, q, s), Q7(d)

EEE: Activities that complement the classroom experience can make learning more meaningful. Students are asked, for example, whether they have had serious conversations with students who have different beliefs or values, or with students of a different race/ethnicity. They’re also asked whether they have studied abroad and participated in student clubs, learning communities, internships and culminating senior experiences.

Survey questions: Q1(l, u, v), Q7(a ,b, e, f, g, h),Q9(d), Q10(c)

SCE: Students perform better when their college is committed to their success and cultivates positive social relationships among different groups of people. Students are asked, for example, whether the campus provides “the support you need” to succeed academically and thrive socially and to assess, for example, the quality of their relationships with other students, faculty and the administration.

Survey questions: Q8(a, b, c),Q10(b, d, e)

Basically NSSE results are not to be used for ranking purposes. McMaster’s use of the results is typical of other Ontario Universities – that is, the Benchmark Report was posted on the University’s web page. Last Spring Macleans collected Benchmark scores from several Canadian universities.

Results from other significant questions

The questions asked in Q#11 of the survey are related to a student’s educational development and personal growth. However the responses to Q#11 do not fit within any NSSE benchmark. Nonetheless, to me at least, they are amongst the most important in the survey as they demonstrate (self-reported) growth in student development as well as the “soft skills” (writing skills, speaking skills, critical thinking skills, etc.) so highly valued by the employers of our students. The term soft skills is politically charged in some faculties – most of us would call them “vital skills” that we expect all of our graduates (regardless of discipline) to demonstrate.

Student Satisfaction Questions

Although the majority of the NSSE questions deal with engagement there are three that measure student satisfaction and two of them are most widely used for comparison purposes. They are: Q#13. How would you evaluate your entire educational experience at this institution? and, Q#14. If you could start over again, would you go to the same institution you are now attending? The results below have been extracted from the NSSE 2006 Mean and Frequencies comparisons found on McMaster’s website.

[ Click here ] to view McMaster’s NSSE Results (a new window will open).

These results were taken from the spreadsheet NSSE06 Mean and Frequency Report (McMaster).xls which is posted on the McMaster Website . So, for example, 266 first year student rated their Experience at McMaster to be excellent. In the data above: “Ontario” refers to the COU consortium with responses from McMaster removed. The Carnegie Peers (that McMaster chose are: Auburn University, Case Western Reserve, Emory University, Georgetown University, Loyola University College, Saint Louis University, University of Arizona, University of Tennessee, University of Miami, University of Michigan – Ann Arbor, University of Missouri – Columbia, University of Pittsburgh, University of Wisconsin – Madison, Virginia Commonwealth University, Washington State University, Wayne State University) and NSSE Refers to the entire NSSE 2006 sample.

[ Click here ] for NSSE 2006 Mean Comparisons (a new window will open)

 

I’ll have more to say about NSSE and the other ranking and student satisfaction surveys in future posts.

 

References

Astin, A.A., “What matters in College: Four Critical Years Revisited”, Jossey-Bass, (1993).

Bloom, B. S. Taxonomy of educational objectives, Allyn and Bacon, Boston, MA. (1984)

Burgan, M., “In Defense of Lecturing”, Change November/December (2006)

Chickering, A.W. and Gamson, Z.F., “Seven Principles for Good Practice in Undergraduate Education”, The Wingspread Journal, 9, 2, (1987)

Felder, R. “What Matters in College.” Chem. Engr. Education, 27(4), 194-195 (Fall 1993).

Pascarella, E.T. and Terenzini, P.T., How College Affects Students, Jossey-Bass, (1991)