Netflix Challenge for SCOTUS Prediction?

During our break from blogging, Ian Ayers offered a very interesting post over a Freakonomics entitled “Prediction Markets vs. Super Crunching: Which Can Better Predict How Justice Kennedy Will Vote?” In general terms, the post compares the well known statistical model offered by Martin-Quinn to the new Supreme Court Fantasy League created by Josh Blackman. We were particularly interested in a sentence located at end of the post … “[T]he fantasy league predictions would probably be more accurate if market participants had to actually put their money behind their predictions (as with intrade.com).”  This point is well taken. Extending the idea of having some “skin in the game,” we wondered what sort of intellectual returns could be generated for the field of quantitative Supreme Court prediction by some sort of Netflix style SCOTUS challenge.

The Martin-Quinn model has significantly advanced the field of quantitative analysis of the United States Supreme Court. However, despite all of the benefits the model has offered, it is unlikely to be the last word on the question. While only time will tell, an improved prediction algorithm might very well be generated through the application of ideas in machine learning and via incorporation of additional components such as text, citations, etc.

With significant financial sum at stake … even far less than the real Netflix challenge … it is certainly possible that a non-trivial mprovement could be generated. In a discussion among a few of us here at the Michigan CSCS lab, we generated the following non-exhaustive set of possible ground rules for a Netflix Style SCOTUS challenge:

  1. To be unseated, the winning team should be required to make a non-trivial improvement upon the out-of-sample historical success of the Martin-Quinn Model.
  2. To prevent overfitting, the authors of this non-trivial improvement should be required to best the existing model for some prospective period.
  3. All of those who submit agree to publish their code in a standard programming language (C, Java, Python, etc.) with reasonable commenting / documentation.

The Supreme Court Open Infrastructure Project Meeting

Wash U CERL Meeting

Mike and I just spent a couple days a Washington University’s Center for Empirical Research in the Law for a meeting related to the Supreme Court Open Infrastructure Project. The meeting featured a number of great folks with cool data projects. The discussion was very fruitful and it is clear that the end product is going to offer a wide range of data relevant resources.  We are looking forward to contribute to the project in the months to come!

Google Wave — A Promising Platform for Real-Time Collaboration

communication_collaboration_google_wave_revolution_id793675_size485

Also from the good folks at Google Scholar comes caselaw and patents together with metadata, page tags and a nice “how cited” feature.  Here is the announcement from the GoogleBlog. Useful analysis available at Legal Informatics Blog, Just in Case and Internet for Lawyers. Enjoy!

"Sink Method" Poster for Conference on Empirical Legal Studies (CELS 2009 @ USC)

Sinks Poster

As we mentioned in previous posts, Seadragon is a really cool product. Please note load times may vary depending upon your specific machine configuration as well as the strength of your internet connection. For those not familiar with how to operate it please see below. In our view, the Full Screen is best the way to go ….

Statistical Time Machines

Time_Machines

So, I was a bit late on this … However, it is a really cool idea and thus I want to flag it for those who might have missed it.  As covered over at SCOTUS Blog and ELS Blog, the November 12th Wall Street Journal features a story entitled “Statistical Time Travel Helps to Answer What-Ifs.”  Of interest to legal scholars, Professors Andrew Martin and Kevin Quinn discuss a series of what-ifs including how today’s Supreme Court would have voted on Roe v. Wade … Check it out!

Hustle and Flow: A Social Network Analysis of the American Federal Judiciary [Repost from 3/25]

Zoom on Network

Together with Derek Stafford from the University of Michigan Department of Political Science, Hustle and Flow: A Social Network Analysis of the American Federal Judiciary represents our initial foray into Computational Legal Studies. The full paper contains a number of interesting visualizations where we draw various federal judges together on the basis of their shared law clerks (1995-2004). The screen print above is a zoom very center of the center of the network.  Yellow Nodes represent Supreme Court Justices, Green Nodes represent Circuit Court Justices, Blue Nodes represent District Court Justices.

There exist many high quality formal models of judicial decision making including those considering decisions rendered by judges in judicial hierarchy, whistle blowing, etc. One component which might meaningfully contribute to the extent literature is the rigorous consideration of the social and professional relationships between jurists and the impacts (if any) these relationships impose upon outcomes. Indeed, from a modeling standpoint, we believe the “judicial game” is a game on a graph–one where an individual strategic jurist must take stock of time evolving social topology upon which he or she is operating. Even among judges of equal institutional rank, we observe jurists with widely variant levels social authority (specifically social authority follows a power law distribution).

So what does all of this mean? Take whistle blowing — the power law distribution implies that if the average judge has a whistle, the “super-judges” we identify within the paper could be said to have an air horn. With the goal of enriching positive political theory / formal modeling of the courts, we believe the development of a positive theory of judicial social structure can enrich our understanding of the dynamics of prestige and influence. In addition, we believe, at least in part, “judicial peer effects” can help legal doctrine socially spread across the network. In that vein, here is a view of our operationalization of the social landscape … a wide shot of the broader network visualized using the Kamada-Kawai visualization algorithm:

Here is the current abstract for the paper: Scholars have long asserted that social structure is an important feature of a variety of societal institutions. As part of a larger effort to develop a fully integrated model of judicial decision making, we argue that social structure-operationalized as the professional and social connections between judicial actors-partially directs outcomes in the hierarchical federal judiciary. Since different social structures impose dissimilar consequences upon outputs, the precursor to evaluating the doctrinal consequences that a given social structure imposes is a descriptive effort to characterize its properties. Given the difficulty associated with obtaining appropriate data for federal judges, it is necessary to rely upon a proxy measure to paint a picture of the social landscape. In the aggregate, we believe the flow of law clerks reflects a reasonable proxy for social and professional linkages between jurists. Having collected available information for all federal judicial law clerks employed by an Article III judge during the “natural” Rehnquist Court (1995-2004), we use these roughly 19,000 clerk events to craft a series of network based visualizations.   Using network analysis, our visualizations and subsequent analytics provide insight into the path of peer effects in the federal judiciary. For example, we find the distribution of “degrees” is highly skewed implying the social structure is dictated by a small number of socially prominent actors. Using a variety of centrality measures, we identify these socially prominent jurists. Next, we draw from the extant complexity literature and offer a possible generative process responsible for producing such inequality in social authority. While the complete adjudication of a generative process is beyond the scope of this article, our results contribute to a growing literature documenting the highly-skewed distribution of authority across the common law and its constitutive institutions.

Citation Analysis in Continental Jurisdictions

Citation Analysis

Anton Geist has posted Using Citation Analysis Techniques for Computer-Assisted Legal Research in Continental Jurisdictions to the SSRN.  While this is certainly longer than most papers, we believe it offers a good review of the broader information retrieval and law literature.  In addition, it offers some empirical insight into citation patterns within continental jurisdictions. The findings in this paper are similar to those shown in important papers by Thomas Smith in The Web of the Law and by David Post & Michael Eisen in How Long is the Coastline of Law? Thoughts on the Fractal Nature of Legal Systems. 

In our view, the next step for this research is to determine whether the pattern does indeed follow a power law distribution.  Specifically, there exists a Maximum Likelihood based test developed in the applied physics paper Power-law Distributions in Empirical Data by Aaron ClausetCosma Shalizi and Mark Newman which can help adjudicate whether the detected pattern represents a highly skewed distribution or is indeed a power law.

Either way, we are excited by this paper as we believe comparative research is absolutely critical to broader theory development.

Institutional Rules, Strategic Behavior, Agenda Control & Inferences — Explaining Chief Justice Roberts Anomalous Decision in NAMUNDO

Punt Team

Agenda Control and Careful Inferences

What are the class of potential inferences one should draw when the Chief Justice behaves in a manner which would appear at odds with our prior understandings of his jurisprudence? As I have argued in my previous article Institutional Rules, Strategic Behavior and the Legacy of Chief Justice William Rehnquist: Setting the Record Straight on Dickerson v. United States, there is significant reason to be careful about the class of inferences one draws under conditions similar to those accompanying yesterday’s decision in NAMUNDO v. Holder.

A significant strain of the literature in political science is devoted to studying the power of agenda control. The primary power of Chief Justice of the United States Supreme Court is the power of opinion assignment. This includes the right of the Chief to assign to himself the task of opinion writing.  Of course, this authority is qualifed as it only applies when he finds himself in the majority coalition.  If he finds himself outside of the majority, the Senior Associate Justice in the majority is permitted to exercise this important authority.  

The opinion assignment norm provides a significant incentive for the Chief Justice to behave “strategically.”  Specifically, in instances where the majority of the court is unwilling to support his preferred outcome, the Chief still has an incentive to join the majority in order to do “damage control.”  For example, he can attempt to author a watered-down opinion or an opinion which leaves the major substantive issues for another day.  

The Ghost of Dickerson v. United States

Consider as an illustrative example, Justice Rehnquist’s behavior in the 2000 case, Dickerson v. United States.  In Dickerson, the Supreme Court was called to consider the ultimate constitutionality of its landmark decision in Miranda v. Arizona. Prior to the Court’s decision, even Miranda’s strongest supporters had expressed significant concern regarding its continued viability. As I sat in the audience on the day of the Dickerson decision, this concern was only heightened when Justice Rehnquist indicated he would deliver the court’s majority opinion.  

Chief Justice Rehnquist prior Miranda related jurisprudence indicated a significant hostility to the Court landmark 1966 ruling. In fact, in every decision prior to Miranda he either voted to limit or undercut the Court’s Miranda doctrine.  In 57 out of 57 prior cases, the Miranda doctrine had no friend in William Rehnquist. Between his decision in Dickerson and his death, the Rehnquist-led Court considered 5 major Miranda-related cases. In each of these cases, the Chief resumed exactly where he left off prior to Dickerson.  He consistently voted to undercut the holding and virtually ignored his own Dickerson opinion.       

Chief Justice Rehnquist’s former law clerk, Ted Cruz, writing about the Dickerson decision in a eulogy in the Harvard Law Review, essentially acknowledged the strategic nature of the decision “it was the best that could be gotten from the current members of the Court.” From a doctrinal perspective, his decision was fairly opaque. For example, in responding to questions regarding Dickerson’s logical underpinning Ted Cruz stated, “do not ask why, and please, never, ever, ever cite this opinion for any reason.”  

The Strategic Decision in NAMUNDO v. Holder?

Nearly four years after the death of Chief Justice Rehnquist, another socially important decision would be surprisingly authored by a Chief Justice who initially appeared hostile to the question at issue.  This time it was Chief Justice John Roberts, a jurist initially socialized in the ways of the high court in the early 1980’s chambers of then Justice William Rehnquist.

In yesterday’s decision in NAMUNDO v. Holder, Chief Justice Roberts authored an 8-1 decision. Leading election law scholars including Professor Rick Hasen have initially described it as “an interpretation of the Act virtually no lawyer thought was plausible.” The lesson from Dickerson and other such cases is strategic behavior on the part of the Chief is always possible. Once it is apparent he does not have the requisite votes to reach his desired policy outcome–what is a strategic Chief Justice to do?  Do damage control, limit the core holding or as Professor Gerken has characterized yesterday’s ruling, “punt.”  

Visualization of the Ideological History of the Supreme Court

USSC MQ Scores

Here is a cool visual for the Martin-Quinn Scores. For those of you not familiar, the Martin-Quinn paper and “MQ Scores” represented a significant breakthrough in the field of judicial politics. On that note, Stephen Jessee & Alexander Tahk have done a nice job both bringing their data up to date and extending their work.  For those interested, click on the visual above and check out all of the relevant links contained within this post.  

An Exchange in Need of Empirics and an Analytical or Computational Model


section 5

On a recent flight, I read Jeffrey Toobin’s New Yorker Article on Chief Justice Roberts entitled “No More Mr. Nice Guy”.  The exchange quoted above is drawn from this article. While I believe it is appropriate to engage empirical data where available, the underlying discussion is not one exclusively subjectable to empirical inquiry.  Rather, it is, at least in part, a question in need of a formal theoretic model. Justice Roberts and Mr. Katyal are implicitly discussing a “state of the world” not yet realized but which would be realized if the statute were not to exist. What would benefit the discussion is principled manner to adjudicate between these two inferences drawn above.  Namely, it would be useful to fully evaluate what behavior would likely follow if the statute were not to exist.

There exist a variety of mathematical modeling techniques which could inject some much needed rigor into the above discussion. To my knowledge, such an applied model has yet to be offered.  The Supreme Court’s decision in the matter is soon forthcoming. Given the nature of the exchange above, there is reason to believe that if Chief Justice Roberts prevails ….we will get our model as the “state of the world” discussed above will no longer be hypothetical…. 

Judge Sonia Sotomayor ⇒ Justice Sotomayor?

Justice Sotomayor?

Justice Souter’s recently announced retirement has generated significant speculation regarding the potential nominee President Obama might select.    

Barring some unknown skeleton in her closet, if President Obama seeks to (1) select a Federal Court of Appeals Judge and (2) increase the diversity of the Court on multiple dimensions …. well … Judge Sotomayor would have to be the frontrunner.  

The picture and graph statistics pictured above are drawn from our paper Hustle and Flow: A Social Network Analysis of the American Federal Judiciary.  In the paper, we offer a mapping of the social topology of the American Federal Judiciary.  Built upon data aggregated over the Natural Rehnquist Court (1995-2004), we find Judge Sotomayor holds a position of significant social prominence.  

To read more on operationalization, etc.—click on the slide above or click here.

A number of commentators have suggested President Obama might forgo nominating a sitting judge — instead choosing an academic or politician.  This is certainly a possibility and in that vein let me reveal my bias in favor of Gov. Jennifer Granholm (for whom I formerly worked). 

Justice Souter to Retire….Possible Replacements?…Here is a Mapping of Socially Prominent Jurists in the American Federal Judiciary

American Federal Judiciary

NPR’s Nina Totenberg is reporting that Justice Souter is planning to retire at the end of the current Supreme Court Term.  As noted in the NPR report, the short list of replacements may include Elena Kagan, Diane Wood and/or Sonia Sotomayor (who is in the network above near Justice Stevens).  If President Obama decides to look beyond this early short list, he might consider one of the socially prominent federal jurist mapped in above visualization. We have a much more detailed prior post on the underlying paper Hustle and Flow: A Social Network Analysis of the American Federal Judiciary located here.  To see the full visualization contained within the paper, click on the slide above or click here.       

Taking Judicial Content Seriously–Lupu & Fowler's Strategic Content Model

Roe v. Wade Citation Network

In my conversations with judicial politics scholars, many lament how many of our existing approaches tend to ignore opinion content.  For those interested in embedding opinion content into existing theories of judicial decision making … consider Yonatan LupuJames Fowler’s paper recently posted to the SSRN.  

The authors present a strategic model of judicial bargaining over opinion content.  They note … “we find that the Court generates opinions that are better grounded in law when more justices write concurring opinions.”  To generate the specification for “grounding in law” the authors use Kleinberg’s Hubs and Authorities Algorithm calculated at the time the opinion was authored. The Strategic Content Paper is available here. 

The visual above is drawn from a related Fowler project located here.  Another very worthwhile paper authored by FowlerJohnsonSpriggs, Jeon & Wahlbeck is located here.