Data Integration Methodologies

Data integration solutions can be categorized into three main forms.

•     ETL - Extract,Transform,Load
•     ELT - Extract, Load, Transform
•     EII - Enterprise Information Integration (Data Virtualization)

   
   
ETL (Extract, Transform, and Load)

• Extract: All processing required to connect to various data sources, extract the data from these data sources, and make the data available to the subsequent processing steps.

• Transform: Any function applied to the extracted data between the extraction from sources and loading into targets.

    Ex:
        Movement of data
        Validation of data against data quality rules
        Modification of the content or structure of the data
        Integration of the data with data from other sources
        Calculation of derived or aggregated values based on processed dat
       

• Load: All processing required to load the data in a target system.

ELT (Extract, Load and Transform)

• The data is first extracted from the source(s),loaded into the target database, and then transformed and integrated into the desired format.All the heavy data processing takes place inside the target database.
• ELT tool needs to know how to use the target database platform and the specific SQL dialect being used.

EII (Enterprise Information Integration)

• Instead of physically integrating data, it is virtually integrated, making the data accessible in real time when it is needed.
• data is always up-to-date.
• There is no extra storage layer and no extra data duplication.
• EII tool takes care of all the translations and transformations in the background.

Enterprise Integration Architecture(EAI)

During my research on EAI, I gathered following facts about it.

Integration of the applications and the business processes is a top priority for many enterprises today. Requirements for improved customer service or self service, rapidly changing business environments and support for the mergers and acquisitions are major drivers for increased integration between existing “stovepipe” systems and information silos. Many different tools and technologies are available to address the complexities of integration of the systems and EAI is one of the most challenging architecture.

EAI is defined as the use of software and computer systems architectural principles to integrate a set of enterprise computer applications. It is an integration framework composed of a collection of technologies and services which form a middleware to enable integration of systems and applications across the enterprise. EAI is a business need to make diverse applications in an enterprise including partner systems to communicate to each other to achieve a business objective in a seamless reliable fashion irrespective of platform and geographical location of these applications.

EAI is a flexible and cost effective system integration strategy and architecture that replaces traditional point-to-point integration and EAI can be described as the task of getting independently designed applications to work together. Before the advent of EAI, applications were interfaced on a point-to-point basis. Interfaces were programmed for each interface point in each application. This solution delivered inflexible interfaces that organizations found difficult to manage. Integration complexity increased as more point-to-point interfaces were implemented. Although this is a tightly coupled and operationally effective approach, it often required significant maintenance cost and can limit the extensibility of the solution in relation to other business applications.

In implementing EAI solutions organizations have been able to realize various benefits including:

• Ease of Development and Maintenance
• Enhanced performance and reliability
• Implementation of a centralized information bus
• Extension of the legacy system lifecycle
• Reduced time to market
• Real time information access among systems
• Streamlines business processes and helps raise organizational efficiency.
• Maintains information integrity across multiple systems

Although EAI is beneficial for a company in several ways it has some disadvantages as below:

• High initial development costs, especially for small and mid-sized businesses (SMBs).
• EAI implementations are very time consuming, and need a lot of resources.
• Require a fair amount of up front design, which many managers are not able to envision or not willing to invest in. Most EAI projects usually start off as point-to-point efforts, very soon becoming unmanageable as the number of applications increase.

References:

1. www.mincom.com/company/whitepapers/eai_whitepaper_us.pdf
2. www.liquidhub.com/docs/Horizons_EAI.pdf