EasyVote Solutions Cloud Leak[This sad looking, but fact-filled page is intended for journalists, investigators, and the public that demands to know more]In January, 2022 it was discovered that EasyVote Solutions, a company that provides software to help manage elections (but is not involved in the election process), had a cloud leak. "Cloud leak" is the term used when a company uses the cloud to store sensitive data, but it gets exposes to the public (normally due to a misconfiguration). EasyVote stored over 209,000 files on the Microsoft Azure cloud service, and allowed "anonymous public read access" so that anyone on the Internet could list and download the files, with one search engine listing over 209,000+ files. Anybody viewing one of those results in the search engine who clicked on one of the link would -- as you may imagine -- connect to Azure's cloud service, and since EasyVote authorized Azure to allow anyone access to the file, it would do so. The exposed files included sensitive data (from SSNs and drivers' license numbers to photocopies of passports and birth certificates) about poll workers in at least 10 states.
Who Are You and Why Are You investigating this?I'm nobody, really. I'm not a security professional. I'm investigating this because I have a friend that does a lot of good for a lot of people, and he is now likely under investigation by the FBI, and would not be if it were not for what EasyVote did. And media reports after the news broke were misleading: e.g. saying "the breach didn’t involve [SSN's]" (incorrect), that they "disabled access to that server" (made up), that they were a "victim of hacking" and "was hacked" (technically false; there was no hack).
Was This the Work of a Hacker?No!The breach -- if it can be called that -- was the result of EasyVote allowing public access to sensitive files. Specifically, the sensitive files were stored in a "container" (the term for a cloud directory/folder) named [waiting for final fix to release], and EasyVote allowed public access to that container by design. EasyVote did have a safeguard in place to make it virtually impossible for the public to guess the names of the sensitive files. However, not only did they allow public access, they also allowed the public the get a full list of all their files. By graning the public the ability to list all their files, and to download them, the breach was there. All someone had to do to access the files was click a link in a search engine.
Who Discovered This? A Hacker?It was not a hacker. The leak was discovered by an older gentleman, who was using a data search engine (just as a regular search engine indexes public webpages, a data search engine indexes public files on websites). He volunteers his time to help people, and as part of that looks for public voter files (in most states, they are a public record, and can be obtained for non-commercial uses). He found some Georgia voter files (not realizing that they were not intended to be public). He then noticed that the source of the files had 200,000+ files made public. He thought there might be more public voter files, but instead found documents with SSNs. He got mad that EasyVote had this data public, downloaded a sample as evidence, and contacted his congressman. He then contacted Congressman Jim Clyburn of Beaufort County, SC (where many of the files were from), and the next business day the FBI and SLED (South Carolina Law Enforcement Division) started investigating.
Was a Hacker Involved In Any Way?Yes and no. There was someone who claims to have downloaded all the 200,000+ files, and has offered them for sale. This person is separate from the person who accidentally discovered the public files.Whether someone selling sensitive files that were offered publicly is hacking goes beyond what I wish to try to interpret.
What Search Engine Was UsedI am not going to post the name of the search engine until either I hear from EasyVote that the issue is completely resolved, or I believe it has been.A "data search engine" is like the search engines you are familiar with (Google indexes webpages), except that it indexes data files (usually in the cloud). Google also indexed at least one file from the container than sensitive files were in. The link I saw was a blank licensing agreement, presumably there so customers could download it before purchasing the software. I do not think that would be considered sensitive data. Google doesn't know how to go through cloud data file lists, so it only had a link to a page that it already knew about (likely because the link appeared on the EasyVote website).
More Details on What Caused This?EasyVote, like many other companies, uses the cloud to store files (using Microsoft's Azure offering, similar to Amazon's AWS S3 service). Azure offers 3 settings: private access (the default, where EasyVote would provide a list of authorized users), public access to files (EasyVote authorizes everyone to download any file without authentication), or public access with file listings (same as public access to files, but you also can get a list of all the file names).EasyVote offers a product EasyCampaignFinance, which by design offers public access to public files. With the data in the cloud, it is much easier to just have public access to that container. So we know that EasyVote was choosing to use public access to the container that the sensitive files appeared in. And we know that container was set to allow public access since at least 2018 (although perhaps without allowing file listings). I'm guessing that EasyVote intended for the sensitive files to have public access. The first problem was that EasyVote had all the 200,000+ exposed files in the same Azure container (rather than separate containers), which meant that the files were either all private or all public. So if EasyCampaignFinance files were public, everything was. However, this alone was not the cause of the leak. EasyVote took precautions to make the file names impossible to guess (using long strings of random numbers called GUIDs to identify customers and files). In my opinion, that's a bad practice, but it worked for them. The second problem was that the files were not encrypted. Had they done this, any exposed files would have only been accessible if hackers spent significant resources to try to access them. The third problem is that EasyVote set the access to the container to allow public access with file listings. That is what did them in. That allowed search engines to see what files were there, and return them in results. The moment they did that, the sensitive files were exposed. So it was a combination of 3 different issues that caused this (and yes, that caused my friend to probably be a subject of an FBI investigation). The cloud leak would not have occurred if they used separate containers (one public, one private), OR if they encrypted the files OR if they did not set the access to allow file listings.
Can You Explain In Layman's Terms What Happened?If you don't know much about the cloud, access levels, authentication, and encryption, here's a fairly simple (albeit absurd) analogy:Let's say a business keeps track of paper records of a county, including public records (campaign disclosure documents) and private records (hiring documents). They have a building open 24 hours a day, just marked "Easy Vote, LLC", with a sign saying "Come in and look at records!" Journalists and curious members of the public know about this, and occasionally go to read campaign disclosure documents. But, there's a well hidden door, and behind it are unlocked filing cabinets with the private records. The way EasyVote intended for this to work, the analogy makes sense: to access the private records, you need to know that the hidden door is there (or snoop around knocking on the walls, but even that likely wouldn't work and would get you caught, this is a very hard to find door). But, EasyVote (in the analogy) opened the secret door and put up a sign saying "More Records Here." With the "More Records Here" sign, people know there is something else (with no indication they are private files); with the door wide open, they aren't snooping -- it's just another room in a public building. And with no lock (encryption) on the cabinets, people can access the files. How Many States/Counties Were Involved?EasyVote stores files for over 250 counties and cities in at least 20 states. But lots of those customers do not store sensitive files with EasyVote. In Georgia, about 120 of their 159 counties store various files with EasyVote. In South Carolina, EasyVote stores various files for at least 40 of their 46 counties. In Texas, at least 29 counties of their 254 counties use EasyVote.Sensitive documents were exposed for poll workers in at least 10 states, and at least around 15 counties. By my calcuations, over 100 countries and cities stored potentially sensitive files with EasyVote. That is how many counties and cities used (I believe) the EasyPollWorker module to upload files to EasyVote. However, many of them may not have had any sensitive files stored.
What is a 'Sensitive' File?I consider a file to be sensitive if it contains a social security number, bank account number or driver's license number (I do not consider date of birth to be sensitive, as it is public in voter lists in many states).Besides those numbers just printed on a form, there were also lots of photocopies of drivers' licenses, passports, and some birth certificates.
Leak? Breach? Hack?This was a cloud leak. A cloud leak is when sensitive files are placed in the cloud, and due to a misconfiguration (rather than hacking) are exposed to the public.Hack? No. Nearly all definitions of "hack" involve a person intentionally trying to bypass some sort of security. That wasn't the case here. Breach? Maybe. Most definitions of "breach" (e.g. CIST refer to unauthorized access. But EasyVote authorized Azure to allow public access to the files. If there was a breach, it was when EasyVote allowed public access (not when the files were accessed). Leak? Probably. Some definitions of "leak" refer to unauthorized access, but looking at the non-computer terms "breach" (intentional) and "leak" (accidental), leak seems to be a much more appropriate term here. Cloud leak? Yes! This seems to be the best possible term for what happened. By adding "cloud", it qualifies "leak" in a way that gives a better idea of what happened, and suggests (to the informed) that this was not an intentional act.
Security Through ObscurityEasyVote's big mistake was relying too heavily (completely!) on "Security Through Obscurity" (STO) -- the thought that by hiding things, they won't be found. Experts tend to agree that STO is a useful addition to normal security, but not great on its own.In EasyVote's case, it appears that early on they made the decisions to [1] have public and private files in the same container, and [2] authorize anyone with a filename in that container to download the file (Azure's public anonymous read access to blobs). So if the anyone is allowed to access the public files, they are also allowed to download the private files if they know the name. Since the names of the private files were impossible to guess, this is a prime example of STO. In theory, their design should have worked. Most customers had a directory name using a GUID (or 16 random bytes), and most files were likewise protected. Assuming the random number generator is reasonably accurate, it would be impossible to guess the filenames. Literally. Impossible. As in taking trillions of years. Someone would have to [1] guess the directory (impossible), [2] guess the filename (impossible), [3] know Azure the storage account name, and [4] know the Azure container name. But, EasyVote gave away #3 and #4 -- they intentionally offered public access to public files using the storage account name and container name that the sensitive files were in. So by design, someone was authorized access to sensitive files if they knew the impossible-to-guess directories/filenames. But, EasyVote made a massive mistake: they set up the access to their storage accounts and containers so that they were not just public, but so that all the filenames could be listed. Once that was done, their only layer of security (STO) was taken away, and the files were there for anyone to see.
Hacking and AuthorizationMost definitions of hacking require it to be "unauthorized" or "without permission". The United States Computer Fraud and Abuse Act (CFAA) doesn't use the word "hack" but is the main law used to convict hackers -- and it (normally) requires that the act be "without authorization" or "exceeding authorized access." So if someone is authorized to do something, it is pretty much by definition not hacking.If EasyVote had these files on their own servers, and they had a URL "http://easy_vote.com/getfile?filename=SOMEFILE.jpg", and someone kept guessing filenames, that might be considered hacking. Why? It's pretty clear that EasyVote isn't intending for you to download files other than the ones they link to. There was a case where someone went to prison for something like this (despite being reversed on appeal for a technicality). If EasyVote kept the default settings of private access, access would have required a username and password that the public would not have, and therefore access would have been unauthorized. If these files were on a website with rate limiting, someone trying to download all the files would get stopped after a handful of attempts, and be presented with a CAPTCHA screen to prove they are human. Bypassing that could be considered hacking. If these files were encrypted, it would likely require hacking to extract the files. However, anyone downloading any of these files connected to a Microsoft-owned computer, and EasyVote gave Microsoft the explicit instructions to allow anonymous public read access. There was no need to provide a username and password, because EasyVote authorized everyone to have unlimited access to those files. In fact, Texas state law "insists on public access" to some of the files in that container, and EasyVote says its software "makes all available filings instantly available to the general public." That functionality would not have worked properly if the container did not have anonymous public read access. By making no distinction between files intended for the public (campaign disclosures) and sensitive files (HR files with SSNs), either both are authorized, or both are unauthorized. The main difference is that one appears in a link on an EasyVote customer site, the other appears in a search engine. And if an individual violates the CFAA, wouldn't the search engine be violating it as well?
What Counties Use EasyVote?EasyVote does not provide a list. However, from the directories that were leaked, along with logos that were also leaked, the following appear to all be customers of EasyVote:
AL: Morgan County EasyVote Inaccurate or Misleading StatementsFirst, I want to be clear: any company that faces a breach is likely to try to "spin" it in their favor.
(C) Copyright 2022 R. Scott Perry |