Email Archives, Are they an Email Roach Motel?

                Email archives originated to solve two basic pain points back in the middle nineties. The driving concern was the serious mailbox size limitations of Exchange 5.5 and earlier.  They earliest archives focused on moving older and large messages into the central archive to control mailbox sizes and preserve the user experience. The second challenge came from regulated financial and trading institutions needing to comply with NASD 3010 and SEC 17a preservation and retrieval requirements. These rules forced the banks and trade firms to invest in specialized Write Once Read Many (WORM) style hardware and software solutions.

                The financial market drove the early development of email archives and shaped the assumptions of what an archive should do and how it should store the messages. All of this concern around defensibility and compliance pushed the developers to lock down archived items in immutable, proprietary compression formats. Worse, since compliance purchases were driving the sales, the storage platforms engineered for the archives focused on read/write performance and single instance physical storage and practically ignored the retention management issues that this created.

                Ten years later, the reality is that the dominant, established products have the storage and compliance issues covered, but are weak on lifecycle management and extraction of business intelligence from this repository of the sum of corporate knowledge. Worse, for those corporations that invested early in a system that is clearly not meeting their business needs, many are now looking at literally hundreds of terabytes of email that cannot actually be deleted on any kind of rational, defensible basis. The WORM storage they chose utilizes item collections or cabinets to keep performance reasonable, but individual items cannot be deleted or expired.

                Smaller companies can learn from the early mistakes of the big boys. Most are not regulated and do not envision retaining all communications forever. Few companies really need to keep business records (beyond the unavoidable tax and public filings) beyond the usual business cycle for their industry. This can be calendar quarters for an online resale site or decades for a mining company. The important thing is to have plan and process that enables employees to keep and access what is critical to their jobs and the company, while shedding non-business messages.

                Given the number of innovative new archiving products and services that have surfaced in the last year, you will want to look carefully at how your communications are stored. Most federal and many state entities are required to store public records in an open source format to make sure that the information is not lost if a software provider goes out of business. Estorian's LookingGlass Interactive Archiving delivers on this requirement by storing the items in Microsoft Outlook MSG files and then zipping collections to effect up to a 65% compression as compared to the Exchange Store size. Even the LookingGlass Spherical Indexing is addressable directly from SQL, keeping the entire solution accessible in case of migration resulting from a merger or acquisition.

                So when you are shopping for a solution to your growing messaging environment, remember to make sure that the solution you pick will not turn into the retention nightmare down the road. Ask the hard questions about retention management, migration strategies and storage formats.