Now we play the rating game

Maclean’s has released its first ranking of Canadian law schools. The magazine’s choices for the top law faculties I’ll leave till later in this post, because there are rather more interesting things going on here.

Let’s first look at the methodology. The schools are ranked equally on faculty quality (50%) and graduate quality (50%), the latter broken down into elite firm hiring (25%), national reach (15%) and Supreme Court clerkships (10%). Here’s the ranking’s director, Professor Brian Leiter of the University of Texas at Austin Faculty of Law, speaking about the breakdowns:

The primary reason for the 50/50 weighting is to make it easy for a reader to disentangle the relative contribution of each element. Someone who has the traditional view that the excellence of a school is a function of the excellence of its faculty can see fairly easily how the schools perform on that dimension; someone who is mostly concerned with how the bench and bar view the graduates can disentangle that consideration simply. … Since the rankings aim to make the raw data more easily digestible, we needed to make some judgments about the relative importance of different kinds of professional achievement.

I count two “easily”s, one “easy” and one “simply” there, and in truth, there is a certain back-of-the-envelope charm to be found in the data-gathering process. For instance, faculty members were measured on citations in Canadian law journals. Why not journals elsewhere in the world? “I can see no reason why the measure of scholarly excellence in Canadian law schools should be citations outside Canada,” said Leiter. Turns out your reference in the Cambridge Law Journal wasn’t as important as you thought it was.

How did the magazine define “elite law firms”? By consulting another magazine — specifically, Lexpert’s list of leading law firms. Since elite law firms in Canada are not to be found among excellent and profitable six-partner general practices in Surrey, Kingston or Saint John, this works out well, doesn’t it?

I could go on, but I’d rather not. Law school rankings are similar to Most Valuable Player results in professional sports — the process and choice of criteria end up being a bigger story than the results themselves, and it gets old in a hurry.

What’s more interesting is that these criteria are uniformly “objective” — they rely on facts and statistics gleaned from data, rather than, say, interviews with leading decision-makers in the legal, business and client communities. I don’t think that “objective” or “subjective” data sources are any better one than the other, but many rankings often try to incorporate a mix of the two. Maclean’s prefers to let the data speak for themselves, and that’s fine.

I do find it interesting, however, that the magazine’s final rankings — U of T first, McGill second, Osgoode third, and so forth — are not wildly different from the results that other less formal surveys and general professional gossip tend to produce. I’m not saying it’s right or wrong, but it is a lot of effort to get to a place we already were.

There’s one important question, however, that hasn’t been asked or answered here. Maclean’s has ranked Canada’s law schools from best to worst. But best for what? Worst at what? What is the essential function that Maclean’s believes law schools serve? If you haven’t nailed down what a law school is supposed to be or to do, it’s kind of hard to say that some schools are better at it than others.

This particular ranking splits its criteria evenly between professors’ reputation among their peers and certain legal employers’ evident opinions of graduates’ work ability. So if you were to call this a “faculty reputation and graduate hiring” ranking, that would be a more accurate description — with the added caveat that only reputation within Canada, and hiring by a very limited number of legal employers, are counted. If those are your indicators of merit, this is the survey for you.

In fairness to Maclean’s, though, I think that most lawyers, and even many law schools themselves, have similarly blinkered views. National law firm managing partners will tell you in very clear terms what a law school is for; legal aid clinic directors will tell you something very different; Superior Court judges and general counsel and deputy ministers will have their own answers. And, oh yeah — professors and deans have some thoughts on the matter, too. Every law school in Canada is unique and has a different and always-evolving set of goals and visions. But each of us tends to look at law schools through the refracting lens of what we believe lawyers should do and what the law should be, and that colours our perception of whether a school is successful or not.

There is no consensus in Canada about law schools’ raison d’être, and most of us haven’t given the subject the good, hard think it deserves for too long a time. Until there’s something approaching a professional consensus on why we have law schools, what they’re meant to achieve and how that can be measured, I don’t see much point in ranking them.

This post originally appeared as a post at Slaw on September 24, 2007.