60 Terabytes: thats what the gov wanted to store digital conversations

[caligirl]

6 Million conversations per month!
http://www.dailykos.com/story/2005/12/18/52011/042

I can't believe I missed this if it was cross posted here.


INSIDER INFO: 6 million wiretapped conversations per month? Facts from IBM
by harrier
Sun Dec 18, 2005 at 03:20:11 AM PDT

In a quirk of fate, I met someone from IBM who was directly involved in implementing the wiretapping system that is all over the news. It is much worse than the news would suggest.

As some have reported, this is definitely not just NSA. The bidding for the technology to do this project was conducted by members of the FBI and the CIA. During the proposal process, IBM was told explicitly that they were to answer questions, not ask them.

One of the most noteworthy comments was that the Government had specified 60 Terabytes of monthly storage for digital versions of conversations. MONTHLY!

more below...

* harrier's diary :: ::
*

At about 11k per call, that is about 6 million conversations per month. (correct my math if you are better at computers!).

This is enormous.

People in the press are concerned that they are tapping the phones of people without just cause. What they are really doing is much, much worse.

1. In reality, they are tapping hundreds of thousands of lines all the time. This is KGB stuff.
2. Every call is stored digitally and indexed accurately, creating an archive that can be used for political purposes in the future. This is not reels of tape that would require massive effort to file through and might become useless quickly. It is highly indexed information that is being kept in storage. Two years from now, they could look into particular phone numbers to check in on cell and land lines.
3. It is cell and land lines
4. They are running filters on every conversation for Farsi and other languages in real time

This is domestic spying on a massive scale, not just a few wiretaps without a judge. This should be cause for impeachment.


edit: a comment later restates the sheer numbers of calls possibly tapped:
Speech is a few kilohertz wide, so something like 10K per second, so 6 billion seconds, 100 million minutes, of call. At 10 minutes per call, this is 10 million calls a month.

[Inland]

1. Either Bush is spying on a massive scale on US citizens

or that handful of al qaeda associates that he refers to as targets have a FUCKING GREAT long distance plan. With all that time on the phone, who could ever have time to carry out an attack?

[populistdriven]

31. HP's XP 12000 stores up to 30,000 Terabytes - THIRTY THOUSAND TERABYTES

How many of these does the NSA own?

http://www.nth.com/about/hp_array.html

[tuvor]

2. Is "harrier" reliable?

I know d-kos is getting a decent reputation, but me and accepting things at face value don't get along very well.

[tuvor]

15. With all the recommendations, you'd think someone could provide an answer.

We're not going off half-cocked here, I hope.

[caligirl]

21. Looking at his other postings

He seems very knowledgeable about what he writes. You can decide for yourself as others will as well.http://www.dailykos.com/user/harrier/diary

[tuvor]

24. Thanks for taking the trouble.

Much appreciated.

[niyad]

3. 11k per call? gee, I wonder how many other people's 11k I am using.

correct me if I am wrong, but a terabyte is 1,000 gigs, I believe.

this is no way implies any sort of support for this administration and its obscene policies. I just think they are going to need LOTS more storage space.

[Stand and Fight]

7. What's in a terabyte, gigabyte, etc???

You're almost right... It's 1,024 gigabytes actually.

Bytes (8 bits)

0.1 bytes: A single yes/no decision (actually 0.125 bytes, but I rounded)
1 byte: One character
2 bytes:
5 bytes
10 bytes: One word (a word of language, not a computer word)
20 bytes:
50 bytes:
100 bytes: Telegram; two punched computer (Hollerith) cards
200 bytes:
500 bytes:
Kilobyte
1,024 bytes; 210;
approx. 1,000 or 103

1 Kilobyte: Joke; (very) short story
2 Kilobytes: Typewritten page
10 Kilobytes: Page out of an encyclopedia
20 Kilobytes:
50 Kilobytes: Image of a document page, compressed
100 Kilobytes: Photograph, low-resolution
200 Kilobytes: Two boxes (4000) punched computer (Hollerith) cards
500 Kilobytes: Five boxes, one case (10,000 of punched computer (Hollerith) cards
Megabyte
1,048,576 bytes; 220;
approx 1,000,000 or 106

1 Megabyte: Small novel; 3-1/2 inch diskette
2 Megabytes: Photograph, high resolution
5 Megabytes: Complete works of Shakespeare; 30 seconds of broadcast-quality video
10 Megabytes: Minute of high-fidelity sound; digital chest X-ray; Box of 3-1/2 inch diskettes
20 Megabytes: Two boxes of 3-1/2 inch diskettes
50 Megabytes: Digital mammogram
100 Megabytes: Yard of books on a shelf; two encyclopedia volumes
200 Megabytes: Reel of 9-track tape; IBM 3480 cartridge tape
500 Megabytes: CD-ROM
Gigabyte
1,073,741,824 bytes; 230;
approx 1,000,000,000 or 109

1 Gigabyte: Paper in the bed of a pickup; symphony in high-fidelity sound; broadcast quality movie
2 Gigabytes: 20 yards of books on a shelf
5 Gigabytes: 8mm Exabyte tale
10 Gigabytes:
20 Gigabytes: Audio collection of the works of Beethoven; five Exabyte tapes; VHS tape used to store digital data
50 Gigabytes: Library floor of books on shelves
100 Gigabytes: Library floor of academic journals on shelves; large ID-1 digital tape
200 Gigabytes: 50 Exabyte tapes
Terabyte
1,099,511,627,776 or 240;
approx. 1,000,000,000,000 or 1012

1 Terabyte: Automated tape robot; all the X-ray films in a large technological hospital; 50,000 trees made into paper and printed; daily rate of EOS (Earth Orbiting System) data (1998)
2 Terabytes: Academic research ligrary
10 Terabytes: Printed collection of the U. S. Library of Congress
50 Terabytes: Contents of a large mass storage system
Petabyte
1,125,899,906,842,624 bytes or 250
approx. 1,000,000,000,000,000 or 1015

1 Petabyte: 3 years of EOS data (2001)
2 Petabytes: All U. S. academic research libraries
20 Petabytes: 1995 production of hard-disk drives
200 Petabytes: All printed material; 1995 production of digital magnetic tape
Exabyte
1,152,921,504,606,846,976 bytes or 260
approx. 1,000,000,000,000,000,000 or 1018

5 Exabytes: All words ever spoken by human beings.
Zettabyte
1,180,591,620,717,411,303,424 bytes or 270
approx. 1,000,000,000,000,000,000,000 or 1021

Yottabyte
1,208,925,819,614,629,174,706,176 bytes or 280
approx. 1,000,000,000,000,000,000,000,000 or 1024


Source: http://www.jamesshuggins.com/h/tek1/how_big.htm

[adolfo]

4. Interesting History

IBM helped the Nazi's track jewish prisoners in the camps and now they want to track American calls.

[jody]

5. Is there a list of words that will trigger recording and processing? If so

then any group that chose to use one or more of those words in each and every telephone call and email would cause the system to react and quickly crash from overload. :evilgrin:

[Contrary1]

11. "Allah" and "Jihad" used on a massive scale might blow a few circuits n/t

[caligirl]

16. Farsi was mentioned. n/t

[soothsayer]

29. Yep, there's a long list I've seen on DU before. I too think everyone

should work them into every conversation.

[Silverhair]

32. Programs are MUCH more sophisticated than that.

They are waaaay past mere scanning for keywords.

Nor would inviting their attention be a good idea anyway.

[Stand and Fight]

6. Good God... This is far scarier than 1984... Far more. K&R. n/t

[aquart]

9. What does K&R mean?

It's a new year and I'm tired of pretending I know.

[LibertyorDeath]

12. Kick & Recommend = K&R

Happy New Year :)

[aquart]

13. Thank you! Happy New Year to you, too!

[Zan_of_Texas]

23. Aquart -- happy new year and hey!

Zan

[Tess49]

14. Thanks for asking that. I've wondered, too.

[hootinholler]

8. Thanks for that! Kinda backs up my speculation

In This thread.

-Hoot

[sandnsea]

10. Average person uses 600 minutes per month

So 150 or so million Americans would use 90 billion minutes. Is that right? So 100 million minutes of calls is 1%, is that right? I'm not at all sure on my math, only on the 600 minutes per month figure.

[caligirl]

17. True $$cost to taxpayer's is????

[ddeclue]

20. How about let's do it as a VISA ad?

60 Terabytes of monthly storage $1,000,000.00/year
SAIC database and sifting software $25,000,000.00/annually
400 Farsi translators $30,000,000/annually.
2000 Arabic translators $150,000,000/annually.
Listening in on your political oppponents in an election year: PRICELESS.

Doug D.
Orlando, FL

[Silverhair]

63. You are way overpriced on the storage rate.

You BUY 60 TB of hard drive for about $50K.

[ddeclue]

18. Ok.. someone needs to do a parody on a Verizon Wireless ad..

Can you hear me now?

Low Rates $10,000,000.00/month
100 million minutes a month family calling plan**
No long distance
No roaming (or we'll send you to Guantanamo)
International calls free (will pickup the bill if you'll just keep talking...)*
No warrant agreement required.

Now where's my photoshop?



Doug D.

*Includes call forwarding, call waiting, voice mail, arabic-english translation services.
**Calls may be monitored for training purposes and to insure quality customer service (oh and we're looking for spies too).

[ddeclue]

26. OK Just did it and it's hilarious.. where can I post it so that I can link

to it here?

Doug D.

[ddeclue]

27. NSA Wireless... Can they hear you now?..How about Now?

?gr4ymuDB.UiISQuS

Doug D.
Orlando, FL

[ddeclue]

19. As large as that sounds...it's just a drop in the NSA bucket..

You can buy a 500 GB hard drive for a few hundred dollars. 60 TB of storage would cost less than $50,000.00 and there are plenty of commercial sites that have that capacity available.

The NSA probably has at least hundreds of petabytes available to them. That's hundreds of thousands of terrabytes available and they are busy beavers capturing all the traffic they can.

Doug D.

[Mr Rabble]

51. The NSA is not using ATA drives for this.

Get real. 500gb of ATA storage is cheap.

500gb of SCSI storage with RAID5 is NOT.

[ddeclue]

53. When you're the NSA SCSI storage is cheap too...

I'm aware of the difference thank you but it really isn't all that relevant.

These people have budgets that make the CIA look like the Red Cross or the Girl Scouts.

500GB of SCSI storage with RAID5 is still less than $10,000. These people have BILLIONS to buy hard drive farms.

Doug D.

[leftofthedial]

22. if you aren't a terrist

you ain't got nuthing to fere.

go buy something, ya commie.

[Vidar]

25. Equals 72 million illegal wiretaps per year. Heck of a job, georgie.

[Hekate]

28. This being the War on Terra, you know that's pronounced "Terra-bytes"

Bite me!

Hekate
who feels SO safe now

[Silverhair]

30. That is not a big capacity.

Illegal is illegal. Stealing a dollar is stealing, just like stealing a million.

Still, that is a rather small storage capacity given the size of the project, and a fairly limited number of calls, given the vast number of calls made daily in the USA to other countries, as well as internal calls.

60 TB isn't that much any more. Desktop exteral hard drives that can hold over one TB are available. A .5 TB sells for about $350.00. I am surprised that they are even leasing storage for that. The NSA computer power is supposed to be awesome, so much so that 60 TB would be a drop in their big bucket.

Something doesn't fit right.

[noamnety]

33. the Navajo Code Talkers

I have no idea what I would talk about in code, but the whole thing makes me mad enough I'd almost be willing to develop one just to make the NSA's life more difficult.

[iconoclastNYC]

34. It's just ridiculous

A part of me thinks that this is just a boondoggle for contracting. We had stacks and stacks of documents related to pre-911 anti-terrorism intelligence that were never even translated. What are they going to do with recording all these thousands and thousands of conversations? If they don't have some sort of automatic transcribing equipment to turn it into searchable text I really don't see the value in it.

This is the sort of program that should be explicitly outlawed by Congress.

[klook]

35. And with today's digital technology it's easy to edit

recorded conversations and make the speaker appear to say things they never said. :scared:

[ddeclue]

36. NSA Wireless.. can they hear you now? how about now? an ad parody

Here's the full size version suitable for wallpaper..

http://www.democraticunderground.com/discuss/duboard.php?az=view_all&address=132x2346815

Doug De Clue
Orlando, FL

[hunter]

37. RAID arrays are cheap.

Indexing is easy.

Big Brother really is watching you.

[Al-CIAda]

38. .

[leveymg]

39. Able Danger used 2.5 terabytes to map out 6385 possible suspects.

Someone correct my math, if necessary, but a similar system with a 60TB capacity would allow for a matrix encompassing about 150,000 people.

Of course, the algorithms and computers are probably more efficient than they were five years ago.

[ddeclue]

41. Yeah but most of this is just a useless academic exercise..

Have you ever heard of "the six degrees of separation?"

I'm a nobody but I can come up with at least 2 different paths by which I would be related in less than 6 degress in such a "network" of callers to Bill Clinton and also can come with a similar less than 6 DOS relationship to both President Bush and Governor Jeb Bush.

This is computerized guilt by association the same as what Joe McCarthy was doing in the 1950's when he was accusing everyone of being a "communist". The only difference is now we are using computers and accusing orders of magnitude larger numbers of people.

Doug De Clue
Bachelor of Aerospace Engineering, Georgia Tech
Programmer, RF Test Engineer

Orlando, FL

[ddeclue]

42. In fact try this DOS map on for size:

FIVE DEGREES OF SEPARATION...

1) I know State Senator Rod Smith who is running for Governor in Florida (though only through meeting him a few times at political events and talking to him.)
2) As a Florida State Senator, Rod Smith knows Florida Governor John Ellis Bush.
3) Florida Governor John Ellis Bush knows both George W. Bush and George H.W. Bush
4) George H.W. Bush knows the Bin Laden family through his business dealings.
5) The Bin Laden family knows Osama Bin Laden..

Oh My God! That must make me, the Bin Laden Family, the Bush family and Rod Smith all terrorists according to the whole NSA "caller web" notion.

1) I know a person who I will not name here who works locally for the Democratic Party in Florida
2) That person previously worked as an staffer for Ted Kennedy.
3) Ted Kennedy knows Bill Clinton.
4) Bill Clinton knows George H.W. Bush.
5) George H.W. Bush knows the Bin Laden family.
6) The Bin Laden family knows Osama Bin Laden...

Oh My God Again! That must also make me, the Bin Laden Family, the Bush family and Bill Clinton and the unnamed person all terrorists according to the whole NSA "caller web" notion.

Try this game for yourself and see how quickly you can map from yourself (probably a nobody like me to someone like the President or President Clinton or Osama Bin Laden, etc., etc.) it's a fascinating game..

Anything more than 2 DOS from the original suspect is not legitimately worth pursuing in an investigation and you usually don't need computers for that. Anything beyond that is just a guilt by association witch hunt and shouldn't be allowed.

Doug De Clue
Orlando, FL

[Silverhair]

47. Depends on the intersections of the web.

There is much much more to what they are doing than mere DOS.

[ddeclue]

48. Of course but it still is rather suspect and not an appropriate use of

our gov't to engage in witchhunts.

I am an engineer and a programmer and have a fairly good idea of the types of stuff they would do to map out people but it still violates the concepts of "probable cause" because they are in essence tossing everybody in America's house FIRST and deciding on suspects SECOND. That's horse before the cart and not allowed in our Constitutional democracy.

Doug D.

[Silverhair]

57. True. I was merely talking about capability, not about the morality. NT

[ddeclue]

49. I've read that they were mapping out to 10 DOS..

and even their own analysts were calling it ridiculous overkill and beyond creepy.

Doug D.

[Silverhair]

58. I seriously doubt that their analysts are saying ANYTHING in public.

The NSA is far more secretive than the CIA. I would believe very little that I hear or read about what they are doing.

[leveymg]

43. My favorite Will Smith movie, along with Enemy OTS. We are all

suspect of something, or we don't count for anything.

This is different than McCarthyism. The President's advisors weren't indicted, then - they are the real target of investigation, today.

One, two, three. They're all going to be taken down, in different ways, all of those who were involved in the OSP-AIPAC and Plamegate conspiracies. Sharon's being forced out, too. Several Saudi Princes and Pakistani Air attaches were also early retired.

The NSA has tapes on them all.

[ddeclue]

45. Well I'm not saying that President isn't guilty...

I'm quite sure he is along with most of his advisors especially in the whole wiretapping situation.

That is not the point I was trying to make however.

I was just using Bush in the DOS examples to show how truly flawed this whole guilt by association methodology is and how much of a threat it is to our civil liberties.

What I'm saying is that the methodology they are using to hunt for "terrorists" is the same as the method McCarthy used to hunt for "communists" - guilt by association network mapping but this time using computers and they are extending it to levels unimaginable to McCarthy by virtue of technology.

Doug D.

[leveymg]

50. What I'm saying is that those who would misuse this power have

more to fear from it.

[ddeclue]

52. Not so long as they control it as is the case now.

The NSA is Bush's power tool the same way that the IRS and the FBI were for Nixon.

Doug D.
Orlando, FL

[leveymg]

61. The Joint Chiefs control the NSA

They hold the keys to the tape vaults. Not the President (particularly this one), not the Secretary of Defense (ditto).

[Javaman]

40. Life during wartime...

"We got computer, we’re tapping pohne lines,
I know that ain’t allowed"

the talking heads were way ahead of their time...

brilliant song, I suggest y'all listen to it again or at least read the lyrics. Life imitates art.

"This ain’t no party, this ain’t no disco,
This ain’t no fooling around
This ain’t no mudd club, or c. b. g. b.,
I ain’t got time for that now"

[Kolesar]

44. Heard of some gravesites, out by the highway

I'm getting used to it now.
One of my favorite jams...

[DainBramaged]

46. Where would they find people to listen to it?

[Old and In the Way]

54. Does anyone think this kind of system is needed to track a few thousand AQ

operatives, financiers, and organizers? How much processing power to track 10,000 people? Probably a fraction of what they have. I'm sure we collect data on lots of other people in this world. Who gets to assign those who are monitored and, more importantly, who is not? How much access and control does the WH have on domestic spying? When does national security become personal security?

[greiner3]

55. Don't forget about email storage! n/t

[rateyes]

56. From wikipedia...

"The books in the U.S. Library of Congress contain approximately 20 terabytes of text."

http://en.wikipedia.org/wiki/Terabyte


So, the govt. is asking, using 3 times the amount of info in the Library of Congress every MONTH? That's a hell of a lot of information...wonder if they have heard any good jokes or recipes?

[Silverhair]

59. 60 TB is a trivial amount for them.

You can buy a 1 TB external hard drive for under $1K. 60 TB simply is NOT a huge amount of storage ability for a commercial use.

[The Straight Story]

60. true, we use petabytes worth on our network

TB's just ain't where it is at anymore :)

[high density]

62. That's what I was thinking

Wal-Mart has some 460tb of data.

[myrna minx]

64. IBM would NEVER do anything nefarious!

:sarcasm:

http://www.ibmandtheholocaust.com/
IBM and the Holocaust is the stunning story of IBM's strategic alliance with Nazi Germany -- beginning in 1933 in the first weeks that Hitler came to power and continuing well into World War II. As the Third Reich embarked upon its plan of conquest and genocide, IBM and its subsidiaries helped create enabling technologies, step-by-step, from the identification and cataloging programs of the 1930s to the selections of the 1940s.

http://www.amazon.com/gp/product/0609607995/103-9130371-1048663?v=glance&n=283155

Editorial Reviews

Amazon.com
Was IBM, "The Solutions Company," partly responsible for the Final Solution? That's the question raised by Edwin Black's IBM and the Holocaust, the most controversial book on the subject since Daniel Jonah Goldhagen's Hitler's Willing Executioners. Black, a son of Holocaust survivors, is less tendentiously simplistic than Goldhagen, but his thesis is no less provocative: he argues that IBM founder Thomas Watson deserved the Merit Cross (Germany's second-highest honor) awarded him by Hitler, his second-biggest customer on earth. "IBM, primarily through its German subsidiary, made Hitler's program of Jewish destruction a technologic mission the company pursued with chilling success," writes Black. "IBM had almost single-handedly brought modern warfare into the information age virtually put the 'blitz' in the krieg."
The crucial technology was a precursor to the computer, the IBM Hollerith punch card machine, which Black glimpsed on exhibit at the U.S. Holocaust Museum, inspiring his five-year, top-secret book project. The Hollerith was used to tabulate and alphabetize census data. Black says the Hollerith and its punch card data ("hole 3 signified homosexual ... hole 8 designated a Jew") was indispensable in rounding up prisoners, keeping the trains fully packed and on time, tallying the deaths, and organizing the entire war effort. Hitler's regime was fantastically, suicidally chaotic; could IBM have been the cause of its sole competence: mass-murdering civilians? Better scholars than I must sift through and appraise Black's mountainous evidence, but clearly the assessment is overdue.

The moral argument turns on one question: How much did IBM New York know about IBM Germany's work, and when? Black documents a scary game of brinksmanship orchestrated by IBM chief Watson, who walked a fine line between enraging U.S. officials and infuriating Hitler. He shamefully delayed returning the Nazi medal until forced to--and when he did return it, the Nazis almost kicked IBM and its crucial machines out of Germany. (Hitler was prone to self-defeating decisions, as demonstrated in How Hitler Could Have Won World War II.)

Black has created a must-read work of history. But it's also a fascinating business book examining the colliding influences of personality, morality, and cold strategic calculation. --Tim Appelo