Reflections on the Competition Appeal Tribunal's Expert Practice Direction on economic evidence

In this article, Nicola Mazzarotto, Lau Nilausen, Michele Avagliano and Soledad Pereiras [1] discuss the Competition Appeal Tribunal's Expert Practice Direction and the concerns underpinning it. They reflect on how to ensure that economic evidence is both independent and proportionate, and how the judicial process could better assist the Tribunal to distinguish between reliable and unreliable analysis, whatever the source of any deficiencies.

The views expressed in this article are the views of the authors only and do not necessarily represent the views of Compass Lexecon, its management, its subsidiaries, its affiliates, its employees, or its clients.

The Competition Appeal Tribunal’s Practice Direction on economic evidence

Economic evidence has much to offer in competition litigation. As the President of the Competition Appeal Tribunal (‘Tribunal’), Mrs Justice Bacon, noted in a recent address, it is “almost always central” to the Tribunal's final substantive decision and private enforcement could not function without expert economists' views on liability and quantum.[2] Broadly, the Competition Appeal Tribunal’s (“the Tribunal’s”) judgments recognise the helpfulness of expert evidence more often than they criticise it.

However, the Tribunal considers that too much of the evidence presented to it falls short of the required standard. Broadly, it has two concerns.

The first is a lack of independence. Economic experts have a duty to be independent, which is held to a demanding standard in UK law.[3] Despite that duty, the Tribunal is concerned that some economists have blurred the line between explaining economic evidence and advocating for their client's position. Illustrating the point, Justice Bacon identified partial evidence, a refusal to abandon disproven positions, and reluctance to engage with alternative views as indicators of the problem.[4]

The second concern is the volume and relevance of evidence. Justice Bacon described expert reports that are habitually “elephantine”, focusing on minutiae and burying key points. That makes it hard for the Tribunal to grasp the real issues and consumes the time and resources of the Tribunal and case teams, driving up costs.[5]

The Tribunal seeks to address those problems through the expectations and procedures set out in its Practice Direction. Broadly, it:

• sets out a case management process to engage earlier on the key issues, questions to be addressed, methodologies, assumptions and data requirements;
• sets expectations on how economic evidence should be presented; and
• outlines how economic evidence will be scrutinised, including concurrent evidence and cross-examination, and the features the Tribunal expects compelling evidence to have.

The Practice Direction is an important and useful intervention. It makes clear for experts, lawyers and parties what the Tribunal expects and finds helpful. If the process for presenting and interrogating that evidence has deficiencies, then it is in everyone’s interests to help improve it. It is also worth noting that this issue resonates beyond the UK. How effectively proceedings enable a court to distinguish between good evidence and poor evidence is an important issue across Europe, even though the specific procedures and details differ.

Here, we offer some reflections on independence, proportionality, and how to improve the process that may further assist the Tribunal to distinguish between quality analysis and unreliable evidence, whatever the source of its deficiencies.

Reflections on independence

Experts lacking independence is a serious issue, needs addressing, but it is not always the root cause of any entrenched disagreement

Experts have a duty to the Tribunal with which they must comply. Quite reasonably, therefore, the CAT is concerned about advocate economists. Experts who appear unwilling to engage with contrary evidence or alternative explanations are a problem for the Tribunal, as they obscure the issues in the dispute. They are a problem for their clients because, as Justice Bacon emphasised, once “the Tribunal gets the sense that an expert is slipping into advocacy for their client, it inevitably becomes harder to trust the rest of that expert’s evidence”.[6] They can also be a problem for the economics profession more broadly. It is one thing for a court to lose confidence in a particular expert. It is another if it begins to lose confidence in economic evidence more generally.

To help address lack of independence, the Practice Direction sets out some formal constraints, such as disclosure of prior involvement by an expert with the instructing client or the case; and “conclaves”, where experts prepare a joint report without the involvement of the instructing lawyers. These may play a useful role, but their impact may also be relatively limited. Such constraints may not be as effective as the Tribunal hopes in forcing experts to “engage in a constructive and realistic discussion, conceding ground where appropriate, rather than maintaining an entrenched position up to the point of trial”.[7] Economists are perfectly capable of entrenched disagreements with one another without assistance from other professions, as any academic conference will demonstrate.

The fundamental issue is whether the judicial process enables the Tribunal to scrutinise economic evidence effectively enough to distinguish reliable evidence from unreliable evidence, whatever the cause of the disagreement may be. Justice Bacon’s observation that advocacy is “self-defeating” for an economic expert prompts the question why the Tribunal encounters the frustrations of entrenched disagreement at all. Two points are worth emphasising.

First, experts acting in good faith can disagree. Economic evidence requires judgement on difficult questions: which assumptions matter, which data are reliable, which model best addresses the issue, and which facts should lead an expert to revise an initial view. Two analysts may take different defensible decisions about samples, variables, specifications, adjustments and counterfactuals. Menkveld and co-authors describe the uncertainty created by these choices as “non-standard errors”, which capture variation in the evidence-generating process itself.[8]

Second, this is exacerbated in litigation as the issues that the Tribunal is called on to assess tend to be novel. The expert community will not ex ante have access to the insights invariably resulting from an issue having been properly tested in precedent. It is therefore important to distinguish between situations in which there are genuine good faith disagreements between economists that the Tribunal by construct is required to arbitrate, and situations in which experts through sins of omission cross the line into advocacy.

That does not mean the impact of advocate economists is negligible – and regardless, they should comply with their duties. They consume time, increase costs, and make the Tribunal's task more difficult. Where possible, those effects should be addressed.

The point is simply to emphasise how important it is that process enables the Tribunal to scrutinise disagreements effectively however they arise. If it does, the Tribunal is better placed to evaluate the evidence, identify potential bias, resolve the disagreements that arise in good faith, and deter advocacy by ensuring the strategy is “self-defeating”.

In that respect, the Tribunal has a good track record, and its process is already in a relatively strong position. Its judgments consistently distinguish evidence it finds helpful from evidence it does not find helpful, and importantly provide its reasons. The Practice Direction itself is a response to the problem that the Tribunal is able to identify. Nonetheless, improvements are possible and desirable, as we discuss below.

Reflections on proportionality

Brevity, consensus, and focus are often products of a good process, not causes of one

The Tribunal may for good reasons be sceptical of reports that are long, technical, adversarial and unwilling to concede anything. Reports that cover immaterial issues and obscure the key issues inhibit the Tribunal’s ability to scrutinise the evidence effectively. Brevity, clarity, and reasonable concessions are desirable outcomes.

In pursuit of those outcomes, the Practice Direction envisages page limits, joint statements, and final trial reports that supersede earlier reports.[9] These proposals are sensible. Constraints such as page limits can impose discipline and priorities on the evidence presented to the Tribunal. That is a good thing. However, it is important to recognise that they are often symptoms of a good process. On their own, imposing such constraints does not necessarily ensure that a good process will emerge.

A short report can be clear because it identifies the point that matters. It can also be short because it avoids complexity. A long report can be wasteful. It can also be necessary because the evidence is complex. Disagreement can reflect advocacy. It can also identify the question that is for the Tribunal to decide.

In this respect Justice Bacon’s address identifies an important point: the proportionality of evidence is often related to its focus, or lack of it. She notes that “vague instructions invite expansive reports with irrelevant material [and] expansive reports invite advocacy”. The lack of precision means that opposing experts have worked from “different assumptions, and sometimes even addressed different questions”.[10] That leads to unstructured debates where the key issues and evidence emerge late in the process. This plays a large role in long, unfocussed reports lacking grounds of consensus. Imposing constraints such as page limits and joint reports does little if that underlying issue remains.

Engaging with the key economic issues early

An important element of the Tribunal’s views is that it seeks to engage earlier, particularly with the Tribunal’s economist panel members.[11] In doing so, it hopes to align the experts as closely as possible on the issues they are addressing and the data needed to address those issues.

Engaging with the key economic issues early could make a real difference. Disagreements can proliferate when experts engage with important structural questions only late in the process, if they engage with them at all. These include issues such as: what is the relevant theory of harm, what counterfactual is being tested, which assumptions are material, and which data can answer the question.

Some economists may leave some of these questions implicit, moving straight to estimation. This is generally not good economics practice and would generally not help the Tribunal. If these structural questions are unresolved, or not addressed, experts will tend to ask for different data, specify different models and address different questions. By trial, the disagreement looks technical and sprawling. In reality, the divergence occurred upstream.

How to ensure that early engagement is effective and practical is a key issue. Part of the solution will require more engagement with fundamental structural modelling questions earlier in the process to the extent possible. But it cannot simply mean asking the Tribunal to engage with the same minutiae, only earlier. Ultimately, experts will need to take a clear-eyed view on which issues are material, how economic analysis may usefully and credibly address them, and which of their differences are rooted in fundamental conceptual issues for the Tribunal to resolve and which relate to secondary issues for which it is not proportionate or meaningful for the Tribunal to reach a definitive decision.

More generally, greater consistency in addressing fundamental structural methodological issues in all cases would also help ensuring that “economic modelling [...] arise from [...] the facts of the case” and that “quantitative economic models [...] be calibrated to align with the observed facts” as well as allowing an expert to be able to better “justify their choice of model” as rightly called for by the Tribunal.[12]

Reflections on improving the process to better scrutinise evidence

Methodological disclosure promotes focus and transparency, not “lock-in”

Through its case management process, the Tribunal expects experts to submit for scrutiny early statements on their directions, the questions to be addressed, the factual and legal assumptions on which they should proceed, the evidence and disclosure on which they will draw, and the methodology on which they intend to rely.[13] This is one of the key mechanisms through which the Tribunal seeks to engage earlier in the economic issues.

The process resembles pre-registration in empirical research. The key motivations are (a) to identify and focus on the most important issues, and (b) prevent analysis that (consciously or otherwise) adjusts its methodology to translate the facts of the case into a pre-ordained outcome. Up-front commitment can guard against bias.

Crucially, the point cannot be to freeze an expert’s approach. That would be impractical and unhelpful. Finding the best methodology depends on the constraints of the data. There is therefore nothing inherently problematic about letting the data inform the methodology, nor even about experts changing their mind. The key purpose is to ensure that there is transparency about what was planned, what changed, and why. Haller and Pereiras make that point in their article on the Damages Directive: pre-registration commits an expert to a process for quantification, not to a specific result or inflexible method, and it requires changes to be explained to the court.[14]

Economics should not be “confined” to purely technical questions.

The Practice Direction says the Tribunal will manage cases actively to ensure that expert evidence is ”strictly confined to the issues for which it is necessary”.[15] That objective is right.

However, that does not necessarily mean economic evidence should be constrained to a narrow set of quantitative questions. The value of economics is often that it helps determine what the right questions are; not only what the answers may be.

This is clearest in the discussion on issues that are mixed questions of fact, law, and economics, all at the same time. Abuse is a good example. Justice Bacon observed that economists will naturally be instructed on market definition, dominance, and quantification. But before they answer whether conduct is “abusive” or “outside competition on the merits”, a prior question must be answered: what exactly is the economic analysis, and is it likely to add anything material, over and above the factual evidence and legal submission?[16]

In many cases, the answer will be that economic analysis has a lot to add on those issues. “Competition on the merits”, for example, concerns the competitive process itself: how firms compete, whether rivals are excluded, whether a strategy is replicable by efficient competitors, and whether consumer welfare is harmed.[17] It is a concept about which economists have much to say (and have said much), even if the final legal characterisation belongs to the Tribunal.

Embracing uncertainty

Justice Bacon said the Tribunal wants an honest “warts and all” discussion of economic modelling, not a “Disneyfied” version that is attractive on its face but not a compelling picture of reality.[18]

The challenge is making it work in practice. And it can, sometimes, be in tension with the Tribunal’s stated desire for brevity. Uncertainty is common in competition cases, as they involve complex markets, strategic conduct, multiple data sets, counterfactuals and effects that cannot be observed directly. That uncertainty and complexity underpins many disagreements the Tribunal must resolve. Even if there is a single issue on which the experts ultimately disagree, that disagreement often depends on a chain of connected assumptions, facts and analytical judgments.

For that reason, it is important for the process to recognise that the value economics brings is not simply estimation. It is a structured framework for identifying those connections and explaining how they affect the conclusions. Often the complexity arises because there are many relatively simple issues layered on top of one another. In that respect, the potential gap between economists and lawyers is not so wide as it may seem; they share a common approach to problem solving. They both interrogate facts against a structured and reasoned framework, to identify the right questions and infer answers. Neither discipline eliminates uncertainty. Both seek to identify it, understand it and decide what follows from it. That provides a common foundation for scrutinising the evidence properly.

In scrutinising the uncertainty of economic evidence it is important that experts do not confuse clear thinking with definitive answers, nor that uncertainty will be perceived as imprecise or muddled thinking. Justice Bacon makes clear that, often, the most useful evidence explains what is known, what is uncertain, which assumptions matter, and how the answer changes if those assumptions change. In that respect, Justice Bacon's call for the “messy reality” of uncertainty has clear merits.[19]

The Tribunal processes for scrutinising economic evidence already provide a good foundation to build on

The Practice Direction seeks to strengthen scrutiny of the evidence. It expects experts to justify their models and specifications, consider alternatives, disclose earlier analysis that materially informed their methodology, and expect questions on their own reports and the reports of other experts. The Tribunal can use teach-ins and concurrent evidence to understand the differences between experts and the economic thinking behind them.

Broadly, in our experience, the concurrent evidence and cross-examination processes already facilitate proceedings well, helping the Tribunal identify the key issues, differences between the evidence, and understand the economic thinking that lies behind them. The hot tubs, for example, in recent cases have allowed the Tribunal to interrogate the issues and separate evidence from speculation. Even in the cases where the Tribunal was critical of some of the evidence; the fact the process allowed it to produce those criticisms is part of its success.

These processes, generally, give a forum for the type of scrutiny and appreciation of nuance required to properly assess the issues set out in the Practice Direction. For instance, where experts must justify their methodological choices and be prepared to consider how alternative models and/or specifications might affect the results, the Tribunal has the opportunity to interrogate whether tests used to assess potential alternatives were credible, or provided thin meaningless robustness checks.[20] Similarly, the Tribunal can scrutinise not only whether an established method was used, but whether it was applied to the right questions in its appropriate context.[21]

That is not to say the process cannot be improved. The earlier it identifies the key questions to which economic analysis can contribute, and the methods by which those questions may be answered, the better equipped courts and tribunals will be to scrutinise the evidence and distinguish between reliable and misleading analysis, whatever the source of its deficiencies may be.

---

References