What is data migration?

Information is an essential asset for any organization and the potential of its value lies in the data that, on occasion, must be migrated to improve the performance of a database, update versions, reduce costs or implement security policies.

But what is data migration?

This process consists of the transfer of data from one system to another and usually takes place at times of transition caused by the arrival of a new application, a change in the mode or storage medium or the needs imposed by the maintenance of the base of corporate data.

Generally, a data migration occurs during a hardware upgrade or transfer from an existing system to an entirely new one. Some examples are:

• Update of a database.

• Migration to or from the hardware platform.

• Migration to new software.

• Fusion of two parallel systems into one that is required when one company absorbs another or when two businesses merge.

In no case should the term migration of data be confused with others that, although similar, show essential differences in the number of sources of origin and destination of data or their diversity. Consolidation, integration or updating of data are different processes with different purposes.

What is data migration, what does it imply and how can it be carried out?

Data migration gets represented by the initials ETL, which correspond to the terms: extraction, transformation, and loading. Although an ETL process can get applied with other objectives, when considering what data migration is, it is inevitable to allude to its primary task: extraction and loading (since the transformation does not have to be applied in all cases, only if necessary).

There are three main options for carrying out data migration:

• Combine the systems of the two companies or sources into a new one.

• Migrate one of the systems to the other.
• Maintain the integrity of both systems, leaving them intact, but creating a common vision for both: a data warehouse.

The most suitable tool to carry out a data migration is one of extraction, transformation and loading, as opposed to less productive options, such as manual coding; other inapplicable, such as the integration of applications (EAI) or others that do not provide everything necessary to carry out the process with full guarantees, as is the case of replication.

To carry out a data migration it is necessary to go through the following steps:

1. Planning– from the definition of the strategy and scope of the project to the feasibility analysis.

2. Analytical– considering variables such as the integrity, accuracy or consistency of the data to be migrated and taking into account the characteristics of the databases of origin and destination.

3. Application selection– can be developed internally or acquired after evaluating the different alternatives.

4. Testing– application of the test cycles to the applications that will use the database.

5. Migration– includes the extraction, transformation and loading stages.

6. Evaluation– it is about measuring the results and analyzing them, determining the necessary adjustments.

Challenges that all data migration must face

Although data migration can be a simple process, its implementation may encounter challenges that will have to be addressed.

• Discover that the source code of the source application is not available and the manufacturer of that application is no longer on the market anymore.

• Find types or formats of source data that have no correspondence in destination: numbers, dates, sub-registers.

• Coding problems that affect certain datasets.

• The existence of optimizations in the data storage format, such as encoded decimal binary storage, non-standard storage of positive/negative numerical values ​​or storage types from which mutually sub-registers are excluded within a record.

• Issues related to the appearance of redundancies and duplications when, at the same time as data migration was carried out, different types of users used the old or the new system or application.

Dismantling the myths

Those who consider what data migration is may find themselves in a difficult position, susceptible to falling into extended beliefs but lacking solidity. Understanding the implications of migration involves discerning the myths that have nothing to do with it:

• Data migration is not a simple process of copying data.

• Data migration is not carried out in one sitting, it is a complex process that has its phases and requires time.

• Data migration cannot be solved only from the outside, it is necessary and highly recommended to have the support of the owners of the data.

• The transformation and validation of data can not, under any circumstances, occur after loading. It must always be done beforehand and the result must be subjected to cycles of tests that demonstrate its suitability to be loaded at the destination.

Better Practices

• Give data profiling the importance it deserves.

• Do not underestimate the data mapping.

• Carry out the profiling tasks at the right time and never after loading.
  
  
• Prefer automatic options to manuals for data profiling.

• Take advantage of data migration to improve the quality of data and metadata.

• Relying on data modeling techniques to optimize integration.

• Keep in mind the operative facet of the data and try to simplify the future user interaction in administrative, reporting or update tasks.