Electromagnetic storeage, the dark side

Post Reply New Topic

Most of what is written down is computer input which is store in some binary encoded scheme. Does this sound good. In a way it does. Digitally encoded material can be stored very compactly compared to ink on paper storage. Here is the dark side; encoding schemes become obsolete in under fifty years (by and large) and much of the material is stored on fragile magnetic material which will degenerate and decompose.

Think about it. The letters that John Adams and Thomas Jefferson exchanged 240 years ago still exist. Some have been photo copied or transcribed to durable paper and ink formats. So we can share the thoughts that Jefferson composed by the light of his oil lamp. The letters written nowadays are e-mail stored where? On disk, on magnetic tape? In what format? Will these formats be readable 100 years from now?

Unless there is a dedicated effort to maintain continuity and translatibility of formats much of what is store now; letters, journal articles, personal memoranda will either decompose or be as unreadable as Linear B or Aztek hiroglyphs.

ruveyn

Last edited by ruveyn on 08 Sep 2010, 1:20 pm, edited 1 time in total.

This is why open standards are a must. Its already becoming a problem for anyone using microsoft office for example. There new docx format is not compatible with older versions of office, this poses a problem if say for example one office sends copys of an important document to another office in a docx format that cant be opened because there not using the latest office version.

Its especially a problem for governments and similar organisations who have to keep records of documents, with only one corporation (Microsoft, Oracle, etc) continually getting government contracts most cases because there is no process for finding alternatives, all government data is increasingly stored using proprietary formats.

And as you say even the encoding scheme for writing data to disk if its a proprietary model the information needed to constrict a program to read such data is hidden away in a company using every increasing copyright protection, patents and corporate secret problems.

Ruveyn is writing about encoding bits of data onto the disk, the actual file format that these bits create is irrelevant. The way data has been encoded on disks has changed gazillion times over the last decades and it also means that it will be almost impossible to retrieve data from older technologies in a few years because there aren't any technologies that could go back and read it. Even one harddrive model changes through its product lifecycle, its controller card changes and the firmware that guides the controller card is also updated.

See:
http://en.wikipedia.org/wiki/Modified_F ... Modulation
http://en.wikipedia.org/wiki/Run_length_limited
http://en.wikipedia.org/wiki/Group_Code_Recording

Anyway, back to the thread: the key to be able to retrieve data is to continuously keep transferring it to a new media format and repeat the cycle every 5 or 10 years. The media format that has been "here to stay" for quite some time is optical disks, i.e. CD and DVD. They can house lots of data - and cheaply too, but they are all very fragile, but then again, you can make 10 copies of each disk and store them in different locations, hell you can even launch a pile of DVD's into space and eventually have a backup of the entire collection of media created by humanity.

But that _IS_ why open standards are needed, the hardware to read anything today be it the encoded bits on a hard drive or a file format wont be around in a few hundred years, the key is to have the blueprint, i.e. the standard used at the time to to allow you to replicate the hardware and software needed to read those bits again.

Open standards are needed for all parts of the chain, hardware standards and specifications, software standards and specifications are both needed if you are to retrieve information from old data storage, many company's keep a lot of this information a secret.

That sounds like the way to go. The entire stored corpus of mankind's knowledge is constantly forward translated to the latest storage media with the latest formats. This would require a constant monitoring of all prior stores of data to make sure they are forward translated.

As far as I know there is no institutional effort currently being made to rescue our stored data past from oblivion. Now most of what is stored is crap, but to make sure the good stuff survives probably all of it will have to be forward translated and later culled.

ruveyn

You would still have to recreate copy's of the CD or DVD's as they dont last forever. they degrade over 10-20 years with high quality dvds within 100 years (apparently).

I thought of a good example. You cant create a copy of something you don't have access to. the voyager spacecraft hold a gold record that includes pictures and sound. The exact point of this topic also had to be solved for the record to have any purpose.

If anyone from the future or from another world got the space craft they would likely not know how to play it or retrieve the information, so information to access the record had to be placed on the record so that anyone could decipher.

This is the cover of the gold record its self which shows the needed information on how to read the disk.
http://upload.wikimedia.org/wikipedia/c ... 001978.jpg

This is an explanation in more readable terms as to what the pictures mean.
http://upload.wikimedia.org/wikipedia/c ... nation.svg

copying data over and over is ok, but there will be situations where data cant be copied, because there is no access to it at the time, or it may have been to late and the hardware and software is no longer available or there may just not be someone to do it.

edit:

a thought. The only reason dead languages are hard to translate such as hieroglyphs is because there is no relational information, the voyager spacecraft for example uses binary and the time period of the transition of a hydrogen atom as a universal base to decipher the information. Without specifications all you get is a jumble of crap.

Argh. I couldn't find a link, but one of the nice ideas from nanotech goes like this...

You have a bin full of "dust" - which is a collection of nice little nanomachines, geared exclusively to recognising (and playing) any recording media.

You throw an old vinyl 33 rpm record in it and it asks you which side to play.

You drop an audio cassette in, and it asks the same question.

You dump your (broken) hard drive in (just the platter will suffice), and it recovers all the files?

You throw an old paperback in, and it reads it to you.

You can even drop the Rosetta stone in (if your bin is big enough) and it will translate it for you.

You are correct in that a digital society will leave little behind for archaeologists to derive much from.

Assuming the current civilizations fall....which they probably will, and archaeologists a few thousand years in the future start to dig them up, they will make one of two conclusions.

The wrong one.
1. There was not much here and humanity was in a "dark" period.

The right one.
2. Our society was advanced and did not store information in a tangible way.

If archaeologists of the future are anything like archaeologist today, the latter option will be the more controversial one.

The best way to ensure they come to the right conclusion is to engrave things on hard rocks and bury them with people.

It's also interesting to note, that in many thousands of years, most of our modern cities will be reduced to shards of glass and lots and lots of ochre as buildings will collapse eventually and their metal frames will eventuallyl rust.

I wonder what will become of the plastic.

I sometimes think this is much ado about nothing.

Anything, with enough time, passes away. The great literary works of history only survived because SOMEONE preserved them. If not for the monks in the Roman Catholic Church, many works would have been lost forever. Someone took on the task of storing, restoring, copying, preserving stuff for posterity.

This will continue. I'm sure optical data storage will be the large winner over time. I'm sure we'll find ways to encode data holographically into glass/crystal in such a way that it degrading over time is essentially a non-issue. The bigger concern would be the fear of some historian/archivist "editorializing" history by choosing to preserve some things and not others....a real issue since the vast majority of data floating out there is worthless but where is the line of "archive quality" going to be drawn and by whom?

Of course, data manipulation is another concern. Not that you can't rewrite the King James Bible for your own tastes and pass it off as history, but there would always be the chance that an unadulterated copy survives and someone finds it to set the record straight. With computer data, such changes are easier to produce and mass-distribute.

And, of course, if technology degrades via some disaster so that we can't utilize the storage device because we don't know how, the knowledge will be lost until some future generation figures out how to read the device in the future. Of course, is that really different from the writings from ancient Egypt which were impossible to decipher before the discovery of the Rosetta Stone?

[quote="Chronos"

I wonder what will become of the plastic.[/quote]

That will last until the sun becomes a red giant and vaporizes earth and everything on it.


ruveyn