The Court of Cassation turns to AI to identify discrepancies in case law

After using artificial intelligence for the publication in open data court decisions, the Court of Cassation decides to use it again to detect differences in case law between the different courts and to check whether the justice rendered is the same for everyone. For this purpose, it called on the scientific experts of the Lab IA, attached to Etalab within the Interministerial Digital Department (Dinum), in particular the ALMAnaCH project team of INRIA. They presented their work in the article “Complex Labeling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings” published on the online platform HAL.

The Court of Cassation is the highest jurisdiction of the French judicial order, its mission is to control the exact application of the law by the courts and the courts of appeal, thus guaranteeing a uniform interpretation of the law.

As a reminder, at the beginning of the year the Ministry of Justice abandoned the Datajust project, aimed at creating an official compensation reference system, thanks to the analysis of case law in the field of bodily injury.

Compare hundreds of thousands of decisions

While the mass of data increases as decisions are rendered by the courts, the Court of Cassation must identify, among the decisions rendered by its six chambers, contradictory interpretations of the same legal question or of a same law. This long and tedious detection work is carried out manually by the Court’s lawyers who have solid legal analysis skills but also a perfect command of the law and case law.

They must first detect similar cases, relying on summary versions of decisions (summaries and key word sequences), also produced manually and are not available for all decisions. There is also a high degree of variability in keyword choices and the level of granularity used.

The Court of Cassation has decided to turn to AI to analyze the hundreds of thousands of decisions and detect all the discrepancies.

Identify discrepancies in case law using AI

Created in 2019, the Laboratory for Artificial Intelligence (Lab IA) supports administrations in the deployment of their AI projects by cooperating with Inria researchers. Ioana Manolescu, its scientific director, thus puts the selected projects in contact with the scientific experts.

The members of the ALMAnaCH project-team: Benoît Sagot, Rachel Bawden, specialists in automatic language processing (TAL) and Thibault Charmet, engineer, worked closely with legal experts and data scientists from the Court of Cassation, for this project.

The identification of similar court decisions can be automated once we know how to automatically measure the similarity between two decisions. The researchers therefore decided to associate judgments two by two on a basis of 80,000 judgments.

To this end, they developed a titration prediction model based on summaries. They assigned a title to cases that did not have one, and then provided additional titles to all cases, on the assumption that this would make it easier to identify pairs of similar documents. To produce these titles automatically, they modeled the prediction of titles from summaries as a machine translation task.

Rachel Bawden, researcher at Inria since 2020 in the ALMAnaCH project-team, comments:

“validating the adaptation of machine translation technologies to other types of data and to other tasks meeting the Court’s needs has been really interesting and opens up prospects for other projects in other areas such as finance or biomedicine.”

A relevant approach

The scientists asked the Court’s lawyers to do the same work as their algorithm after having defined with them a hierarchy of similarity levels. The experiments conducted not only demonstrated that the automatic approach yielded results similar to expert judgments, but also that the additional titrations reinforced these similarities.

The collaboration with the Court of Cassation continues, in order to finalize semi-automatic models to facilitate the drafting of titles by experts. On the side of the Court of Cassation, these final models will be integrated into the workflows of the Court soon, the members of the ALMAnaCH project-team, for their part, are ready to renew the experience with other public structures.

References :

“Complex Labeling and Similarity Prediction in Legal Texts: Automatic Analysis of France’s Court of Cassation Rulings.” LREC 2022 – 13th Language Resources and Evaluation Conference, Jun 2022, Marseille, France. ffhal-03663110f


Thibault Charmet, Benoît Sagot, Rachel Bawden, ALMAnaCH project-team, Inria.
Ines Cherichi, Matthieu Allain, Urszula Czerwinska, Amaury Fouret, Court of Cassation.

We would love to thank the author of this write-up for this incredible material

The Court of Cassation turns to AI to identify discrepancies in case law

You can find our social media profiles and the other related pages