FOIA, AI and the State Department
State Department pilots AI FOIA decision-making
Background
I try to keep the substack on a few distinct topics: FOIA litigation, historically significant disclosures, and even a few practice pointers. AI is not one the subjects, but AI is in everything these days. Recently, the subject of AI came up at the Chief FOIA Officers Council meeting.
For context, the FOIA Improvement Act of 2016 established the Chief FOIA Officers (CFO) Council. The statute envisioned a type of deliberative body that exchanged recommendations for increasing FOIA compliance and promoted common performance measures for agency compliance. The CFO Council is co-chaired by the Directors of OIP and the Office of Government Information Services (OGIS) and is composed of all agency CFOs and the Deputy Director for Management from the Office of Management and Budget (OMB).
As official as the Council sounds, historically the scope is not nearly as exciting. The Council is like a meetup for FOIA Officers who want to exchange ideas, which is often hit-or-miss because convincing government agencies to get on the same page is a lot like herding cats.
Nevertheless, they do occasionally have interesting speakers. This last November, the CFO Council’s main meeting had a speaker from their Technology Subcommittee. Mr. Eric Stein, Chief FOIA Officer from the State Department, spoke about how the State Department is using AI to support their responses to FOIA Requests. As they reference in the post here, Stein’s presentation built off a prior pilot program where the Department of State used AI and machine learning for declassification.
As for why, the State Department has been one of the problem children of the federal world. The data shows that the department’s backlog — defined as the number of FOIA requests “that are pending beyond the statutory time period for a response” — reached nearly 14,000 at the end of fiscal year 2020 and exceeded 16,000 at the end of fiscal year 2021. That was an increase from the 11,000 backlogged requests at the end of 2019.
To illustrate the point, in one of the cases I’m watching Professor Scoville of Yale’s School of Law has filed a lawsuit over multiple year long waits he’s faced in requesting simple diplomatic cables through the FOIA process. No surprise, then, that the State Department is trying to be at the forefront of finding technological solutions.
State Department’s AI FOIA Pilot Program
Unfortunately, there’s not much on the pilot itself on the CFO Council’s webpage. The transcript they have is from the main meeting in November, and while the State Department’s speaker has two slides and twenty minutes at the main council meeting there is nothing from their Technology Subcommittee. Here is the agenda. In theory, the meetings are also posted to the National Archives’ YouTube channel, however I have not found a livestream recording. If any of my readers have better YouTube searching skills / time than me, please feel free to add it in the comments.
There is the following blurb posted to the Department of Justice’s OIP website, which shows that the State Department’s program is building off prior success with declassification.
Conveniently enough, I recall that Stein was interviewed at the beginning of the year by the Federal News Network. From his quotes we can get a good feel of what the program is, and what it is not. He says, “I think people are afraid of AI, and maybe they should be. Maybe they shouldn’t be, but my take is, we’d like to get people comfortable with the concepts of AI and machine learning.” From that we can deduce they are using a type of language model rather than a ‘true’ AI. Unfortunately, while readers should read the whole thing the perspective is prospective because the interview is dated in June.
Now perhaps someone less determined, or with less time on their hands, wouldn’t keep worrying away at the problem. Eventually I did find a presentation provided to the National Archives FOIA Advisory Committee. While the State Departments’ slides do not specifically say it’s the same program, I’d take my chances that there are not two AI pilot programs happening at the same time.
Given that, there are a few things that I want to pull out.
First, FOIA is a booming business. Not only is the State Department getting more requests, but their requests are going to grow in tandem with ever more information that must be reviewed. According to the State Department’s record management schedule, more emails and more (de)classified cables will be available to requestors starting in the next few years.
The increase will also not just be the State Department. As referenced above, the State Department’s statistics are not alone in hitting all-time highs. Last year, the 120 federal agencies subject to FOIA collectively received more than 928,000 FOIA requests — 90,000 more than in 2021, according to the Office of Information Policy. Meanwhile, the number of backlogged requests in 2022 — nearly 207,000 — also reached record territory, up nearly 50,000 from the previous year.
Two, the State Department is turning to a little-known government outfit called MITRE. Note the reference to “FOIA Assistant.”
MITRE is difficult to explain conceptually, but MITRE is a private, not-for-profit corporation that supports federally funded research and development centers (FFRDCs). If you have ever heard of the Rand Corportation, then you have a general idea of what role MITRE plays. They are one of those lesser known but omnipresent government corporations that have a hand in more things than their low profile would suggest, and within the defense cybersecurity federal world they’re everywhere. MITRE isn’t a household name, but there’s an argument they really should be.
Forbes did a little article about MITRE a few years back, and I’ll quote their pithy description. The whole thing is worth reading, and Forbes’ description answers why MITRE is at the forefront of bringing machine learning to federal agencies.
Mitre’s influence goes far beyond its vast tech development; it’s also a major consultant for myriad government agencies on how best to deploy tech and policy strategies. Its latest gig: helping the Centers for Disease Control and Prevention (the CDC) and Homeland Security's ominously named Countering Weapons of Mass Destruction office craft sweeping plans for curtailing the Covid-19 pandemic.
Another example, MITRE currently has dozens of what are called Indefinite Delivery / Indefinite Quantity contracts with the Office of the Secretary of Defense (OSD), the purpose of which is to manage and operate the different scientific efforts. However, you don’t actually do anything with an IDIQ contract; rather, the purpose of the IDIQ contract is to have Task Orders placed on it. These TO’s are essentially mini-contracts in and of themselves, and all the actual work is performed according to the TO’s by the federally funded research and development centers that MITRE administers.
One of these IDIQ contracts is for FOIA support, and those TO’s have provided the impetus for the creation of “FOIA Assistant,” which is the program referenced by the State Department in that slide. Not too much is known about the program except for what MITRE has on their website.
We do know, however, that the State Department isn’t alone. Executive Gov., likely relying on scouring publicly available contract language, reported that the State Department was simply the first to go on record. CDC, Department of Justice and Department of Defense are close behind.
Third, and finally, the State Department’s pilot is really two AI models to help process FOIA requests. One model employs machine-learning algorithms to find records in the agency’s centralized databases and archives, which hold more than 3 billion records. The other pilot sends prompts to people submitting FOIA requests via the agency’s web portal based on the words they input, suggesting where they might be able to find information that’s already publicly available or that they narrow the scope of their requests to help facilitate quicker responses.
Either way, we’ll know more in February when the pilot program is expected to end, and hopefully I’ll have the presence of mind to return to this subject for my readers and we can close the loop on whether the program was successful.
This was really interesting. I’ve been involved in a couple of FOIA processes on the nonprofit side - once in prep for potential litigation, and once in a direct FOIA suit against the USDA for failing to timely release records. I’ve long thought someone could create a lucrative practice based solely on FOIA litigation on behalf of nonprofits. (These may exist in DC or the like, but my legal network is unaware of anyone with such a practice on the west coast.)
They’d have to be pro bono, but the cases are straightforward and could be handled by newer lawyers; the plaintiff’s burden is simple and the big wrangling (at least in my field) is about release schedules and redactions. These take time of course, and fees are typically awarded.
On the other hand, I am well aware that agency budgets for their actual work (like doing science or human services, etc.) are often drained by these types of payouts for litigation expenses (and don’t agencies have to “pay” DOJ for a defense too?). I want our federal agencies to be able to provide the services they were established for, so I can hope that this LLM model being trialled might provide a way forward. (And it sounds like there will be a backlog of old FOIA requests for years to come to keep any FOIA litigators in business.)
"FOIA is a booming business" - this is heartening. I'm less enthusiastic about the AI element but I suppose time will tell. Very interesting as usual. Thanks, Max.