So you’ve bought and installed all your company’s application systems: marketing, sales, accounting, manufacturing, provisioning, shipping, and human resources. Now – who’s going to copy and paste all that data from one system to another? This isn’t want they meant by the man-machine interface – the machines were supposed to serve us, not the other way around! So let’s take another look at valuing data integration properly instead of ignoring the problem, because it won’t go away on its own.
Forces Driving Data Integration
Microsoft defines a few forces driving the growing world of data integration to include:
- The employment of multiple systems generally incompatible with one another.
- Prevalence of presentation, business logic, and data logical layers.
- The need to be noninvasive such as not to interrupt production systems.
- Isolation of internal data structures among application’s internal data structures.
Without data integration techniques and technologies, these needs would be considerably more complex to solve, and human labor would be providing much of the interface between applications. Manual integration of systems is a very short term option that wastes human effort. People are almost always going to be more expensive than machines for repetitive tasks.
Up-Front vs. Long Term Costs
Why do we care about data integration products? Business goals dictate managing costs, and naturally this extends to vendor product licensing and IT labor costs. However, initial cost should not be the only force driving an enterprise choosing a data integration strategy. While cost of implementation is often the largest force driving enterprise decisions, it’s only natural to consider the maintenance cycle costs as well.
Rarely are an organization’s data needs static. As the volume and complexity of data and services integrations rise, an organization will either subscribe to vendor or third-party support solutions, or spend a large amount of money on dedicated in-house support teams. While considering the initial cost to change directions or switch systems is of high importance, other major factors that should be considered are support costs, flexibility, and scalability to identified missions. Just as the bulk of a software system’s lifecycle is spent in maintenance, most of the technical debt incurred from data solutions is spent updating, fixing, and changing the initial solution to accommodate ever-changing business needs and demands.
Complexity and Learning Curve
The data integration product chosen by your predecessor may look great on their resume after their having gone to class for a month and spending a year to master its complex proprietary user interface, but is it something that you really want to deal with? Perhaps you don’t have a month to go to class to learn this beast, and just want to get the job done. The hidden costs of support and the complexity of some data integration products make it far too hard to respond to the needs of the business, and waste time and money. Here’s where you make a career decision: do you want to impress people with your incredibly unique and complex skill set required by an integration tool, or do you want to get the work done quickly, easily, and with transparency?
While many vendors may tout well-polished data integration solutions that claim to save on maintenance costs, the cost and man-hour requirement of training itself must be considered as a component of the total cost to the company. Time spent on training cannot be spent on revenue-generating projects: first-party data integration is a necessity and rarely a source of additional profit, since even end-user benefits garnered from integration is rarely more than a value-added service. The more complex the solution, the more time is required to train and master a proprietary technology, which translates to a higher overall cost. Even teams rolling-out their own solutions can be considered to be burdened with training costs, albeit internally. Some solutions—specifically well-established solutions, and more precisely those cultivating massive followings—may be so complex as to require months of university-level training just to configure the product on-premises. With such training and expertise required, personnel costs skyrocket as individuals can either become irreplaceable, creating “functionality silos”, or potentially be poached for other opportunities requiring the same experience.
Is it Time for a Fresh Start?
So, why would anything besides upfront costs influence an enterprise’s data integration practices or purchases? Newer or smaller businesses or subsidiaries may benefit from looser integration demands, where integration may be more of a value-added benefit than a demand. An organization may be sufficiently “fresh” that they are blessed with small data volume or simple requirements. But larger and mid-size enterprises most often need to support incredible volumes of data spanning several decades. While there are a multitude of vendors supplying data integration products and countless in-house solutions in use across the world, few if any products work directly out of the box with no vendor or organizational intervention or setup.
The initial solution to an enterprise’s integration needs can come in many forms: between in-house solutions and vendor solutions carefully dissected, modified, and frequently backed by first or third party support teams lies a multitude of solution strategies. Most often the largest component of initial costs outside of any applicable licensing is how many man hours will be required to train the appropriate personnel responsible for solution configuration, maintenance, and updates. These valuable man hours frequently create blockers for projects and cause costly delays. Development teams can be over-stressed by the introduction of change. Any changes to corporate data models create spikes in technical debt that can spread to bottleneck an entire organization. These bottlenecks create avoidable costs for companies of all sizes which can cripple entire product lines or the company itself.
Change fuels the flames of enterprise: as customer demands scale, how well does the integration between systems keep up? Does integration power your business or has it become a major bottleneck? Does rigidity paralyze your enterprise from accommodating new ideas and practices? This is where flexibility becomes a major factor: what incremental cost need be incurred to support change? How fast can this change become a reality? All of these questions present major concerns to consider in weighing the costs and benefits of how you can integrate enterprise systems.
When the effectiveness of a solution is measured by total cost and time-to-production, clearly the optimal solution for an organization of any size is one that
- requires the least training
- utilizes familiar concepts
- operates on solid technical foundations
- synergizes instead of bottlenecking
- enhances rather than constricting flexibility of implementation
- minimizes total cost, both short and long term
- is easily sustainable
- requires minimal maintenance