Can’t get no LSATisfaction

Here’s something interesting: the consultancy Kerma Partners recently conducted an in-depth study of more than 1,300 current and past “timekeepers” on behalf of an AmLaw 25 law firm. The study identified which personal qualities and attributes of lawyers correlated most strongly with firm success factors such as productivity and longevity. Lawyers possessing the best of these attributes, it turns out, doubled or even tripled other lawyers’ showings in terms of profitability. The authors don’t reveal what these “best” attributes are — presumably, the firm will keep that information close to its vest – but lawyers’ law school rank and GPA were not among them.

Those results are in themselves serious food for thought. What’s significant about this study, though, is that it happened at all – a very large and successful law firm set out to rationally reconsider all of the assumptions it makes and criteria it uses in its lawyer recruitment process. It’s part of what I’m coming to think is the quiet but very real advancement of empirical analysis in the process of how the legal profession understands itself.

Of course, rational, evidence-based analyses of legal education and legal practice have been around for a while. There are excellent blogs that examine empiricism in the context of law schools (Best Practices in Legal Education and Law School Innovation, to name two) and law firms (Empirical Legal Studies and Adam Smith Esq., to name two more). But just last month, I flagged some emerging studies that cast serious doubts on, respectively, the utility of the LSAT, the traditional criteria for law professors, and the importance of law school grades in identifying and shaping good lawyers.

There are also academic institutions that study how we train our lawyers and run our law firms. Georgetown Law’s Center for the Study of the Legal Profession is probably the market leader. But again, last week also saw the arrival of William Mitchell College of Law’s Center for the Empirical Study of Legal Practice, which aims to “research the daily realities of the work of a lawyer and the business of the legal profession.” The new center is led by political science professor Herbert Kritzer, who quotably describes himself: “I am to lawyers what Dian Fossey is to gorillas.”

Some smart and dedicated people are putting the profession’s habits and assumptions, particularly those relating to who should be a law student and what makes a good lawyer, under the microscope. Not a moment too soon, I say. And at this moment, the magnification is being turned up especially high on that old standby, the LSAT: the increasingly maligned test that figures prominently in many law school admissions systems (and in the equally maligned US News & World Report rankings of US law schools).

Last week, Berkeley Law School released the results of a six-year study that highlights numerous important skills for future lawyers — including creativity, negotiation, problem-solving and stress management. These are attributes that the LSAT simply isn’t configured to detect, but that, if the Kerma and similar studies are any indication, matter a lot to whether you’ll be a good lawyer. Berkeley is now looking for other schools to pick up the thread of its Law School Admissions Project and help fully develop other admissions tests that can at least complement the LSAT and fill in the yawning gaps it misses.

The LSAT also figures heavily in a controversial development at the University of Michigan, which has instituted a new admissions channel for undergrads at U of M. “Wolverine Scholars” will be admitted to the university’s law school if they maintain a 3.8 undergrad GPA — and if they do not take the LSAT. Both the Moneylaw and Empirical Legal Studies blogs think that’s kind of strange — why not simply ignore applicants’ LSAT scores, rather than mandating that students never take the test? Among the possible answers: to increase the schools’ median applicant LSAT score and improve its US News & World Report ranking. If that is in fact the case, then it’s indicative of a serious problem: the tail is now indisputably wagging the dog.

(And by the way: it seems that US News & World Report is following other aging periodicals like the Christian Science Monitor by dropping its print edition and moving entirely to online format. Word is that USNWR will now publish monthly, abandon news reporting altogether, and focus on its rankings, such as those for law schools. The magazine stated that it’s becoming “a multi-platform digital publisher of news you can use and analysis.” You don’t need to see through that kind of corporate claptrap to know that USNWR’s days as a force to be reckoned with in law school assessment are likely numbered.)

I don’t really want to light the torches and lead an uprising against the LSAT here: it does what it does relatively well, and doesn’t do what it was never designed for. The problem with the LSAT is that it’s only a hammer, and we now need an entire deluxe toolbox to solve our problems with the profession’s admission and quality evaluation criteria. We contented ourselves for decades with a law school admissions process that was essentially two-dimensional (LSAT and GPA) and a new lawyer assessment regime that barely expanded upon that (law school grades and law school ranking/reputation). I don’t make predictions as a matter of course, but I feel very safe in predicting that those days are just about gone.

The ironic thing about this development is that the LSAT was meant to be an improvement: an easy measure of applicants’ intellectual capacity, standard across all jurisdictions, that corrected for differences in undergraduate education and other factors. Rather like the billable hour, it was supposed to ensure fairness and accuracy through simple numbers. But not only is simplicity insufficient for our needs, so too is our quaint mid-20th-century faith in numbers — in the application of a formula to a calculator to guarantee superior quality in the legal profession.

William Henderson makes this point at the ELS blog:

When admissions officers are under constant pressure to beat last year’s numbers, something has to give.  I suspect it is students who took challenging majors but have LSAT scores slightly under the target.  Or applicants with impressive work experience, evidence of leadership, or a history of overcoming major obstacles.

Over the years I have talked with many admissions officers. They corroborate the sea change.  Further, many of the old hands argue that the current fixation on maximizing numbers is misguided–that, based on their experience, great candidates are being passed over for nondescript or unadventurous students with high numbers. In other words, a large portion of candidates with compelling resumes and personal statements are being systematically pushed down to lower ranked law schools.

Designing complex systems that more accurately assess the most promising and deserving candidates for law school promises to be a lengthy, costly, frustrating process with many false starts. But it’s going to happen. We might, down the road, find ourselves yearning for the easy days of LSAT and GPA scores – but we can’t any longer pretend they’re the best or fairest gatekeepers to our profession.


Comments are closed.