Where have all the e-mails gone?
March 23 2008
A few weeks ago, I blogged about the US Congressional hearings related to e-mail retention at the White House. The House oversight committee hearing was the latest chapter in a saga of technical ineptitude, political intrigue, and hidden agendas. It wasn't entirely surprising to see politicians (and competitors) try to point fingers elsewhere, anything to try to explain why a potential five million e-mails are lost...and those are just the ones that passed through the official White House e-mail system.
Following the hearing, several Lotus customers and partners contacted me expressing concern over the way that Lotus Notes was characterized in those hearings. The sequence of events that followed that was quite dramatic for me, even after 20 years in the industry -- I ended up on the phone with Congressman Darrel Issa, who could not have been nicer or more understanding of what issues were raised by his comments. I have received a letter from the Congressman, which I hope to publish in the next week or so. The hearing testimony will also receive an amendment clarifying the intent of the commentary about Lotus Notes. (If a copy of that letter would help in any discussions in your organization, let me know and I'll plan on getting it to you when I can) Bottom line, Notes wasn't the focus or target of the hearing, so no research had been done about it in the context of the EOOP (Executive Office of the President) CIO's testimony. Now everyone knows, and we move on.
But where? I've also mentioned here that David Gewirtz, Editor-in-Chief of DominoPower and OutlookPower, recently published a book entitled, "Where have all the e-mails gone?" David sent me a copy of the book a couple of weeks ago, and I couldn't put it down. He's written it in a very engaging tech writer kind of style, but he doesn't go too deep into the bits and bytes of DNS, MX records, or the like. It's all there to support his points, but the main focus is on unravelling the mystery itself.
I enjoyed reading the book, but I couldn't help feeling that this is all still a work in process. No fault of David's, but a book that raises more questions than answers is almost guaranteed to leave you going "hmm". I know Gewirtz is still very much on the case -- having continued to write about it for his publications -- but the answers simply haven't materialized yet.
One point that really helped in David's narrative is establishing how the White House got from point A to point B. There are knocks against almost every US presidential administration going back 25 years, and how they have addressed (or hidden) the issue of e-mail as it pertains to the national archival record. It's kind of depressing, but it is reality -- they are no smarter or better-prepared than even the best IT shop in the country. David's big question is, how much of that is competency and how much of it is deliberate?
Link: Where have all the emails gone? >
Post a Comment
- 2
Danny Lawrence | 3/24/2008 8:43:44 AM
I'd agree with Nathan, nothing against federal IT workers, but the levels of bureaucratic BS that they have to deal with every day, combined with a lack of direct accountability means that most federal IT systems aren't running optimally.
- 3
Keith Brooks http://www.vanessabrooks.com | 3/24/2008 9:26:48 AM
Actually in fairness, the lack of funding, and also salary levels, leads to a "you get what you pay for" world.
I believe they have good people inside, and red tape does not stop you from configuring or administering your servers. What does prevent it is not keeping them up to date because of budget limits.
When I have been in government spaces I am always amazed at how far behind their software, hardware is.
Nothign we can do for them, I would love to help the government out because it's a great gig, but the value to them isn't there. If they were more efficient well you know, that would possibly help the system :-)
- 4
Roger Hintz | 3/24/2008 12:22:11 PM
"David's big question is, how much of that is competency and how much of it is deliberate?"
That is exactly what I was thinking when I first read these blog entries.
- 5
Bill Brown | 3/24/2008 12:27:22 PM
What's really amazing is that each administration can just come in and rip out whatever is there and replace it with a different product that performs the same basic functions.
Where was the ROI for the shareholders (the US taxpaying public) in ripping out Notes and replacing it w/ Exchange? Or replacing whatever they had before that?
- 6
Bruce Elgort http://ideajam.net | 3/24/2008 1:57:26 PM
@5,
How is this different than corporate CIO's :-)
- 7
Turtle http://www.weightlessdog.com/shell.nsf | 3/24/2008 2:10:06 PM
Years ago I was in Notes 4 training with some guys who were contractors and yes, their responsibility was the White House email system. Those guys were on the ball, and if anybody wants to now claim that zillions of emails were just "lost" with a crew that good (and with that many clearances) running the shop, I say that the emails didn't disappear, they WERE disappeared. Deliberately. It takes one hell of a lot of effort to lose things in a well-run Domino shop, which is why I've never lost any of the hundreds of thousands of documents we manage. Cluster replicas, replicas on test or dev servers, tape backups, copies retained on older drives that were taken out of service for upgrade... stuff almost always exists in multiple copies, so if you can't find something, it's because someone consciously wanted it gone.
- 8
Brian Pletcher | 3/24/2008 2:24:43 PM
Line 3328 of
{ Link }
"I wouldn't want to do business with somebody
sti11 using Lotus Notes or still using wooden wagon wheels."
- 9
Ed Brill http://www.edbrill.com | 3/24/2008 2:30:27 PM
@8 Brian, right, this is the portion of the record which is being amended. We discussed it in the prior blog entry on this topic, which I should have linked:
{ Link }
- 10
Danny Lawrence | 3/24/2008 3:27:03 PM
@6 Bruce, at least a corporate CIO is only wasting his company's money (or that of the shareholders and/or customers) The White House wastes the money of all of us taxpayers, there is a difference.
- 11
Steven | 3/24/2008 3:29:23 PM
@2: What is the basis for your statments? Apart from the "...levels of bureaucratic BS", most of your stms would be more accurate if you had said "some" federal IT systems aren't running optimally. Which I'd have to guess is not too differnt than a similar proportion of private sector IT systems. There are also VERY signficant differences between the WH and nearly all other federal agencies, especially when it comes to records managment and retention of email.
- 12
Daniel Lieber http://www.iiui.com | 3/24/2008 10:36:39 PM
The EOP/WH does have different standards than most of the federal government. The EOP is responsible for sending official records to NARA or its designate. Most of the EOP records are permanent whereas very few from most other agencies are permanent and sent to NARA. NARA provides guidance to agencies and the national archivist is being proactive in handling electronic records. There is still much work to be done. With that said, there are, and have been, solutions to the records management issues on both Microsoft and Lotus platforms for several years.
Having the technology available is valuable. Having it properly implemented is of great value. Actually using the proper technology in the right way, priceless!
- 13
Dave Madison | 3/24/2008 11:50:12 PM
"It wasn't entirely surprising to see.....competitors try to point fingers elsewhere,
No kidding!
{ Link }
The implication was clear where you originally wanted the finger pointed.
I'm sure the Exchange team gladly accepts your apology.
It was implementation and system management failures. The email platform was not a major factor, as Al Gore's office can certainly attest.
I agree with Steven @11. The post makes a lot of hasty generalizations about the competence of IT depts in the govt. They are likely more or less reflective of corporate America as well. The EOP just has more visibility...because of the specific person occupying and the apparent need of people to look for sinister motives at every turn.
- 14
Nathan T. Freeman http://nathan.lotus911.com | 3/25/2008 3:58:19 AM
"the apparent need of people to look for sinister motives at every turn."
That is an amazing conclusion to see coming from a Microserf.
- 15
Ed Brill http://www.edbrill.com | 3/25/2008 6:44:32 AM
@13 the book makes it quite clear that switching e-mail systems from Notes to Exchange, in the build-up to the war in Iraq, was fraught with peril. The testimony of the EOOP CIO doesn't exactly answer the question of why it was done. And ultimately, mails were lost as a result of the new system, moreso than any prior administration (Bush the first simply had the backup tapes carted off to his presidential library in the darkness of night as he departed the White House, for example...they're not lost, they're just out of the National Archives). The fact is, if they were still running Notes, which had a working archival system, the story today would be different.
- 16
David Gewirtz http://www.dominopower.com | 3/25/2008 11:18:02 AM
First, big thanks to Ed for his great comments. As you said, Ed, there is more to this story. Since the book was finished back in November, I've done three more articles and am working on another today, in fact for Friday's issue. There's nothing more fun than digging through court documents and congressional testimony!
In this latest round, it's getting more confusing yet. Put briefly, the Magistrate Judge asked the EOP why they couldn't do forensics on their computers and the EOP basically said the computers don't exist any more. There's more to the EOP's answer than that, but I'm still digging through it for all the meat.
This is a disturbing story because at the end of the day, as an IT guy, the "why" question keeps coming up. There are practices I'm seeing that none of us, if we wanted to keep our day jobs, would be allowed to do, even in small IT shops.
I don't like asking "Why?" I'd rather this just be cut and dried, just about the tech. But there's so many little bits and pieces about that IT operation that don't add up quite right.
By the way, this audience, reading Ed's blog and DominoPower, and our fellow IT people, are the ones who are likely to both see the depth of the weirdness here as well as, at least a few of you, are likely to someday wind up having to fix it. So please keep up on this topic and I'm not the only one with the clue stick. Everyone who has insight should write about it, blog about it, and share what they know.
Anyway, again, thanks for the awesome comments.
-- David
- 17
Mark C | 4/3/2008 12:08:27 PM
Well now.
@14 talks about "the apparent need of people to look for sinister motives at every turn."
Those who believe there is no such thing as coincidence will doubtless have not been overly surprised by the US Federal government's suspension action against IBM towards the end of March.
they are no smarter or better-prepared than even the best IT shop in the country.
I think they might be no smarter or better-prepared than even the WORST IT shop in the country.