“Data compliance” sounds like a thing that you should do if you have lawyers and regulatory agencies looking over your shoulder, maybe something that you can put off for a while. You might be in trouble with the boss if you lose a lot of data, but you’re definitely going to be in trouble if that new ecommerce system isn’t in place by the time your company launches its newest product. At what point should you care? When the federal compliance agency fines your company? When that one-time data fix-it script overwrites or deletes massive amounts of customer information?
Let’s take a reasonable approach to data compliance. First what is it?
Information is an asset which must be
- kept private,
- secured from accidental destruction,
- exchanged securely,
- backed up with appropriate retention of all active and deleted records,
- versions of each record kept for auditing and potential recovery,
- rapidly recovered when necessary.
The process for records management governance is to:
- Inventory all organizational records—paper and electronic. Analyze for records retention and legal protection.
- Determine retention policies for each kind of record.
- Ensure regulatory and legal compliance.
- Ensure business continuity.
- Protect your information.
- Make sure your records stay around.
- Get rid of records before it is too late. In some cases, it is mandatory to destroy confidential information after a certain retention period.
- Train the team. Teach everyone about good information handling process and security policy.
- Monitor, audit, and optimize. Learn from your mistakes. Have someone do a penetration test.
To ensure these objectives are met, state, federal and international authorities have put in place a multitude of regulations and laws. In the United States alone, there are in excess of 10,000 regulations which apply to industry-specific private and public companies. The regulations are mandatory and include heavy penalties for violations. Therefore, IT organizations must now operate under these regulations for corporate data retention, historical tracking and recovery.
Originally, the SEC restricted acceptable backup media to either paper or microfiche. This restriction was lifted in 2003 to allow companies to alternately use some forms of electronic media to satisfy regulatory compliance. However, the type of electronic medium and the strict data retention and tracking features associated the electronic medium are of particular importance to ensure compliance.
Key findings from an AIIM survey (see www.aiim.org) found that:
- 71% of organizations have a procedure for retrieval of paper records in the event of litigation, but only 57% have one for electronic records.
- Of those organizations with no Electronic Records Management System, 60% would not be confident, if challenged, that their electronic records have not been changed, deleted or inappropriately accessed.
- 38% of those polled admit that there is little or no enforcement of their records management policies and 55% set no guidance on dealing with important emails as records.
Companies everywhere are evaluating how optimally to address regulatory compliance for their cloud-based customer data. The evaluation criteria are the flexibility, simplicity and cost of deployment and management of the compliance options. All of these factors must also drive a solution that fits seamlessly into the existing corporate IT environment and simultaneously capitalizes on the in-house skills of the technical team.
Likely Scenarios Requiring Data Recovery
Just because you have implemented a back-up plan, will it will be a simple feat to recover and restore your data? Many of us have been shocked and disappointed when the inevitable happens. It’s no secret that all companies at one time or another have outages, operating system failures, human errors, and lost or corrupt data. For a complete fault tolerant disaster recovery plan, companies are now depending on sophisticated solutions to protect valuable corporate information. A January 2013 report from the Aberdeen Group showed that 32% of the companies that are using SaaS services have reported losing data. Job-threatening scenarios involving data loss or corruption can happen to anyone, and can cost your company serious loss of customers, future business, or operational capabilities.
We hear concerns about what if your Cloud application vendor suddenly loses their data center or has a catastrophic data loss. It can happen. The question usually involves how you would continue operations using the functionality of the application. That generally is not feasible, as what it would take to provide even a simple read-only user interface is beyond the scope of most IT departments. However, the most typical scenario is not a Cloud application vendor, losing your data. It is someone within your organization losing it due to one or more of the following issues:
- Malicious employees
- Careless employees
- Employees covering up mistakes
- Administrative errors (mass destruction)
- Misuse of vendor tools (mass destruction)
- Update errors by automated processes
- Inability to trace who did it
Companies who need data compliance should rely on third party solutions. Building a home-grown solution is not cost effective, and application vendors don’t spend a lot of effort giving you the same capability that you’d need to export your data and turn off your subscription.
You may be able to get away with a 14-day retention period for backups, although it seems that it always takes one day longer than the retention period to discover and resolve data loss! That said, a typical requirement for a public company or one in a regulated industry is to
- Back up all data.
- Keep all deleted records.
- Retain the backup for seven years. This can mean keeping the entire application system as a daily snapshot, or versioning the individual records. Storage requirements are must more significant for snapshots, as that implies 2,557 copies of the entire dataset (365 days x 7 years, plus one or two leap year days). Plus, snapshots only contain one version of a record, which may have been updated several times in a given day.
- Be able to recover lost or corrupted data in a time frame that doesn’t significantly affect operations.
Data Recovery Requirements
This should be obvious, but it is not. There are many so-called “compliance solutions” that are missing critical functionality:
- Be able to recover corrupted data – field-by-field – to any point in time. Perhaps a user changed a working phone number to an obsolete phone number, merged records and kept obsolete information, tried to cover up a mistake in a business process. Perhaps a batch process overwrote thousands of records with bad data.
- Be able to recover deleted data. There are many ways to delete data records, either by users deleting individual records, by system administrators deleting thousands of records with a single mouse click, or batch processes going awry.
- Be able to recover the structure of the data, including
- Parent-child relationships (Company has Contacts)
- Recursive relationships (Company has subsidiary Companies)
Many companies have audit requirements that involve research into all the history of any given record. This can be accomplished by taking a complete snapshot of all the data, or by keeping a log of changes to individual records, or even by logging individual fields on records. The history could be a prior image of the entire record, or a change log of each field updated showing who changed it, when they did it, and the values before and after the change.
Whether your business applications are on-premise or in the Cloud, take compliance issues seriously. Loss of job is just as easy as loss of data, and you’ll never be sorry you took too many precautions. Key components are
- Back up all electronic records
- Keep all versions for an appropriate time period according to regulatory, legal, and business needs
- Be able to trace the life cycle of a record and it’s changes
- Be able to rapidly recover anything to any point in time