Welcome to DU! The truly grassroots left-of-center political community where regular people, not algorithms, drive the discussions and set the standards. Join the community: Create a free account Support DU (and get rid of ads!): Become a Star Member Latest Breaking News General Discussion The DU Lounge All Forums Issue Forums Culture Forums Alliance Forums Region Forums Support Forums Help & Search

Erich Bloodaxe BSN

(14,733 posts)
Sat Mar 14, 2015, 09:33 PM Mar 2015

So let's talk about email.

I was in a thread recently in which it was fairly obvious that people were looking to reporters and spokesmen from the administration as 'authoritative sources' on what is and isn't possible to do with email. The problem was, it was fairly obvious that nobody who works with computers was being consulted by either the reporters or the administration spokesmen. So how does email 'work'?

Computers do some pretty amazing things, and they do it by building tiny basic functions up into bigger, more complex functions. I'm going to skip a few layers of explanation that probably would only confuse most folks, and start midway up, at a point where what we think of as 'data' or 'files' are sort of 'readable' by humans, rather than talking about 1s and 0s and registers and gates and so on.

So what are some of the basic things computers do? They take input, like the mechanical key presses you do on your keyboard, they store it in various ways, they manipulate it, and they move it from place to place, sometimes within a single computer, sometimes handing it off to other devices. They also have clocks, and perform tasks at set times using those clocks, sometimes in millisecond increments, sometimes on human scales of hours or days or months. When you look at a screen, the image is refreshing constantly, more quickly than your eye can detect, at set intervals, with updated information if anything has changed.

Ok, that's all well and good, so what about email? Well, an email is just a stream of data that's been stored and formatted in a standard way, so that other programs can decide where it needs to 'go', and how to 'decode' it when it gets there. When I type into any email program, all I'm doing is giving a small processing program the basic message I want to send, and some of the information about who I am and who I want to send the email to and so on. That processing program then appends and prepends other bits of information to properly format my email, and then places it somewhere that another piece of code will find it, verify that it's properly formatted, and then move it on it's way. When it arrives at its destination, it's stored where the person who wants to read it can use another program to do so.

So far so good, right? Ok, now there are several ways to 'store' email. You can keep it in what's called a 'flat file', so that each email is a separate file in a folder somewhere, you can spool it together in a single (or several) big long file(s) along with other emails on the system (all of them, for a single 'user', or for some other combination of circumstances, depending on how your mail system works) or you can store it in a database. If a person knows where to look, and has the appropriate permissions in the system, they can actually go directly to that file and read it using any sort of text editor. (For the sake of simplicity, I'm not going to talk about extra layers of encryption here.)

In every case, though, that email exists in one or more files 'somewhere' until a command is issued to delete it, either manually, or by an automated scheduler. It might be on a central server, or it might be a local machine. But because it's really just another file or set of files in a directory on a machine, a small program (what is called a 'shell script') can be written to make copies of those files elsewhere, whether on the same machine or a different one. That script can be set to simply check the directories in which email is stored, look for any new activity since the last time anything was backed up, and copy anything new off to the 'archive' directory.

This isn't even really an 'email' function, although many email servers and programs have built in mirroring or archiving or whatever you want to call it. It's simply a basic functionality that exists in every major operating system, and is used all the time by the operating system itself for various other reasons. (An 'operating system' - windows, linux, unix, etc - is a set of programs that interact with each other to make the computer 'work'.)

So if some reporter is telling you that the government servers 'couldn't' back up email, the reality is that that reporter simply doesn't know enough about how computers work, and what can be done with a very simple script written by most competent programmers who understand a bit about scripting for their particular operating system. If some administration official is telling you 'it couldn't be done', they're actually telling you that they don't understand how computers work, and how they've worked ever since operating systems were first designed. You don't need to buy any 'extra software', although, again, most email software has such functionality already written into it anyway.

They're not 'lying' to you. They're just not talking about something they actually understand, and making incorrect assumptions about what 'can be done' by computers and programmers, and not actually talking to anyone who knows what they're talking about in IT. They might even have a basic understanding of computers, and just be assuming that unless the email programs have 'archiving functionality' built in, they "can't do it". But they're simply wrong, because any such back-up can be done at a much more basic level, external to any 'email program' or email server code.

5 replies = new reply since forum marked as read
Highlight: NoneDon't highlight anything 5 newestHighlight 5 most recent replies

Thinkingabout

(30,058 posts)
1. I read the .gov servers was not backing up the data, not that it could not be done.
Sat Mar 14, 2015, 09:53 PM
Mar 2015

There was a directive which went out in 2011 and to be completed by 2016 to make sure all .gov servers was backing up. Backups has been around for years but as for myself I have not had a regular backup occurring everyday, week, etc and this did not mean I could not backup but I did not back up. There is a difference.

Erich Bloodaxe BSN

(14,733 posts)
2. Congrats, then, you read it more right than some.
Sat Mar 14, 2015, 09:56 PM
Mar 2015

Others in the thread kept going back to statements made that made it seem that the servers were 'incapable' of backing up files, that the 'only way' they could be backed up was 'manually'. Which really only tells us that no one had asked a competent IT person to set the email directories to be backed up without manual intervention on an email by email basis.

DemocratSinceBirth

(99,708 posts)
4. I heard something crazy in passing..,Maybe I misheard it...
Sat Mar 14, 2015, 10:09 PM
Mar 2015

Out of one billion State Dept e-mails only 60.000 are archived.

 

cascadiance

(19,537 posts)
5. sendmail that is used to send mail doesn't just have messages stored on the destination host...
Sat Mar 14, 2015, 10:30 PM
Mar 2015

Sendmail, a UNIX process that sends mail between machines, first sends it from the host that often sends mail to an interim "relay" host (and these in older days could be multiple relay hosts that it cascaded through), and then gets sent to a destination host where the destination computer pulls its mail from, if it's not the same machine. So though mail is likely only stored permanently on the destination host, it is forwarded and lives in files on other machines while in transit. Depending on how these relay hosts are configured, they could also be looking or saving off the mail too, though that isn't done normally. My guess is that having her own domain used to host email let Clinton configure more where the relay hosts were that in between her and her own computer that she could have controlled more too.

Perhaps the reporter is thinking that relay hosts typically quickly delete mail messages after they are forwarded and validated as sent to their destination to say that normally such mail isn't and not easily "backed up" on one of these relay hosts, which he might have interpreted as mail that "couldn't be backed up". But it really is dependent on how these rely hosts are configured.

I suspect some of these government relay hosts may be configured to store off such mail for the NSA in ways we perhaps aren't publicly aware of that perhaps has sophisticated analysis software scanning such emails that do quick notifications to who knows who for their spying activities. Now one would like to think that if it is just regular email sent/received by people not officially on surveillance, that at most the files are just saved off and not scanned immediately and only looked at later if there's a valid court order from FISA or some other court to do so for certain individuals. But we don't really know. And evidently many like Hillary Clinton, and even Dianne Feinstein who is a fan of government spying, but didn't like hearing that she was being spied on in recent senate statements responding to the Snowden inquiries, are afraid of them being spied on without being under suspicion for anything, and by people working against their interests, perhaps in improper fashion as well.

I think word is out, though not very public to the rest of us, that many congress people and other government officials are looking for private email solutions to avoid this kind of potential spying on them. They don't want to make this widely public outside of their circle, since it will likely have the PTB behind some of what is likely unethical spying activity respond with blackmail using data that they know about these targets through their spying activities.

This in my mind is far more important to tackle at this point, and I'm more inclined to believe that this is the source of why Hillary Clinton privatizing her email rather than anything that unethical in her emails themselves. Now some of the emails may be more exposing of her agenda that she publicly acknowledges that many of us who are concerned about them being more corporatist in nature might dislike if we were able to hear about them, but I suspect there isn't anything there that is her committing any real criminal acts.

As long as there aren't any concerns supported by evidence that might have the NSA working with a warrant to spy on her actively, or others like her, she and others should feel secure in how they communicate with others through private email, even if some of those mail contents have data that reflects opinions and plans, etc. that may be legal, etc. but what many in the public don't like. If people using mail validly and not engaged in illegal or unethical conduct as public servants, then they should feel secure in what they are doing from scrutiny by others that want to for political or other unjustified purposes try to expose what could be sensitive emails from foreign leaders, etc. for their own personal gain.

If people in government service feel that they have to use private email to engage in normal communications in ways that its privacy isn't abused, then there is something wrong, and that should the number one problem that we all look to resolve in these email controversies. That problem isn't personalized to Hillary Clinton or any other single politician, but a general one and the way the system is set up that needs fixing. If this is the problem, then someone who's aware of it who speaks up about it and speaks out on the need to fix this is the kind of person I want to look at voting for higher leadership in our country. That person is someone that wants to help shelter those that work properly for us, and not shelter those that would use our infrastructure for corrupt purposes and criminal activities.

And who watches the watchers is critical in this equation in my book.

Latest Discussions»General Discussion»So let's talk about email...