Assessing Results
Posted: May 28, 2011 Filed under: Uncategorized 3 Comments »What gets measured gets managed.
Peter Drucker
Figures often beguile me, particularly when I have the arranging of them myself; in which case the remark attributed to Disraeli would often apply with justice and force: ‘There are three kinds of lies: lies, damned lies, and statistics.’
Mark Twain
It is difficult to measure results in many legal matters.
When I was general counsel, attorneys would come into my office and trumpet a great resolution: a case dismissed, a terrific settlement , on occasion, a judgment in our favor. We celebrated, but should we have? For instance, when is a dismissal a victory? To answer that question honestly, at a minimum, a determination needs to be made as to whether a similar result could have been achieved less expensively and more quickly.
Then there is the challenge of assessing the merits of a settlement. Settlements are typically measured against the assessed risk of a matter. Lo and behold, most settlements I have seen come right in under their assessed risk. So, are we really able to measure a result that way?
They say a good settlement is one where both sides are unhappy. I once called our CEO to tell him we had negotiated a “good” $400 million settlement. His reply was short and (not so) sweet, “Listen @#X%*, it is never “good” to pay $400 million.” Assuming the plaintiff reacted similarly, perhaps this case met the mutual unhappiness metric.
We also need to ask, was there a way we could have resolved the matter before proceeding to court in the first place? Could the business principals have sat down and talk sans lawyers? So many times, not enough energy is devoted up front to resolve a matter. The business people try, give up and punt it to the lawyers, who blithely proceed to court and it takes a long time before the resolution ball lands back on the field.
In addition, shouldn’t we assess the manner in which the matter was resolved? What was the relationship between our client and the client on the other side? Was that relationship preserved, restored or destroyed? How was our approach to this dispute perceived in the media? By our customers? By regulators?
We need to develop metrics that allow for the results of resolution to take into account the full picture, not simply the amount of money that changes hands.
What metrics do you use?
Whatever the metric, I think that it needs to include ‘disputes avoided’. Companies need to invest in preventive law. Economists likely would tell us there is some optimal amount of it. Spend too little on prevention and the result is too much spent on resolving disputes. Spend too much and likely you are over-investing. I recognize it’s one thing to say this, another to figure out a metric to capture it.
Once a dispute does arise, I suggest that systematic use of decision trees are the best way to measure outcomes. It’s true that many litigators feel decision trees are more trouble than they are worth but I am not aware of a better or more systematic way to capture both risk and costs. It will take some creative work to factor in to the decision tree the dollar value of impact on relationships. But absent a systematic approach that forces lawyers and business people to quantify risks as percentages and estimate dollar values, it’s not clear to me how else to paint the full picture.
Rich — we do an “after action” at the conclusion of every matter. In fact, our matter management system requires the managing attorney to input the results of the after action/lessons learned before the matter can be closed. We focus on 2 elements: procedural — what did the team handling the matter do well and what can be improved; substantive — what processes and procedures should the company keep or adopt to avoid the problem in the future.
As to metrics, at the matter level, among others, we measure deviation from expected value (what did the decision tree say this matter was worth); budget to actual spend; and cycle time. At the business unit level, among others, we measure legal spend as a percentage of revenue; total dispute costs; and implementation of lessons learned. We also watch trending data for number of training sessions, hotline incidents, safety incidents, and warranty costs.
BTW — we agree with Ron on the use of decision trees — the discipline inserted is invaluable and tends to focus activities on things that really contribute to success. An added benefit is that they permit lawyers to talk to MBA and engineer business folks in a language they actually understand.
Trying to find a credible metric for evaluating litigation outcomes is something of a fool’s errand. Too many variables. For example, in most instances, settling a case for $95 that has $100 in exposure is a crappy settlement. But what if that case is sucking a ton of management’s focus so execution is impeded? What if the existence of the litigation imperils a strategic transaction, like a merger? While there no doubt is a way to quantify such factors, my bet is any effort to do so would be a form of voodoo economics where the result will be twisted according to whatever outcome bias you enter it with. I think evaluating litigation outcomes is like the oft cited quote about pornography: [the lawyers] know [a good settlement] when they see it. And, for the record, the $400mm settlement was good.