Data Migration
Let’s jump in and learn:
What Is Data Migration?
Data migration is the process of moving data between two or more systems while changing the format, storage, database, or application. The process of data migration is one of the most important considerations when planning any system implementation, consolidation, or upgrade.
Data migration has four main categories: storage migration, application migration, and business process migration. Regardless of the type, data migration requires that data be prepared, extracted, transformed, and loaded into another system.
This process is referred to as ETL (i.e., extract, transform, load). While the transform step may not be required, data migration always includes the extraction and loading steps.
There are a number of drivers for data migration, including:
- Adding data-intensive applications (e.g., database, data warehouse, data lake)
- Adoption of a cloud-based data platform
- Application or database migration
- Data center relocation
- Data consolidation after a merger or acquisition
- Implementing a new application
- Infrastructure maintenance
- Migrating from an on-premises data center to the cloud
- Moving to a new database or data warehouse
- Server maintenance
- Server or storage equipment replacements or upgrades
- Software upgrades
- Transferring data to third-party cloud providers
- Website consolidation
Data Migration Challenges and Risks
As with any other complex process, data migration challenges and risks must be addressed in advance to avoid errors and delays. Even a small project requires a thorough plan. Taking the following steps before undertaking a data migration project can reduce data migration challenges and risks.
- Break the data migration down into smaller parts rather than undertaking a move of all data at one time
- Contact key stakeholders in advance to secure buy-in, identify potential issues, and incorporate special requirements
- Ensure that the team has the experience and expertise to implement and manage the data migration, and bring in external resources if not
- Establish roles of those who have permission to create, approve, edit, or remove data from the source system
- Have a clear understanding of what data needs to be moved and why
- Prepare a plan for testing and maintenance of data after the migration
- Validate and test data before undertaking the data migration
A few specific challenges that are common with data migrations are:
- Business logic in the form of stored procedures and triggers are not supported by the target system
- Different character sets—for instance, PC-based platforms use ASCII encoding, and mainframe systems are mostly based on EBCDIC encoding, which is incompatible with ASCII
- Lack of interfaces to access the data, which is experienced with older applications
- Unmatched data types (e.g., number, date, sub-records)
Types of Data Migration
There are five main types of data migrations.
- 1. Storage Migration
Backups for disaster recovery or other purposes are moved:
- To a new location
- From paper to digital
- From hard disk drives (HDDs) to solid-state drives (SSDs)
- To a new data storage medium, such as the cloud or tape
- Generally, the catalyst for these types of migration is the need for technology upgrades. In the case of storage migration, the data usually remains unchanged.
- 2. Database Migration
Database migration can be homogenous (i.e., upgrade to the latest database management system, or DBMS, version or a new DBMS provider), or heterogeneous (i.e., switch to a new DBMS from a different provider, such as from MySQL to PostgreSQL or from Oracle to MS SQL). Depending on the target database, the data format can vary and require a transformation step.
- 3. Application Migration
Application migration involves moving a software application, as associated data, from one computing environment to another (e.g., from an on-premise computing system to a cloud platform or from a public to a private cloud platform). With application migrations, the data usually requires transformation.
- 4. Business Process Migration
Business process migration requires the transfer of databases and applications containing a variety of data, including data related to customers, products, and operations. Data often requires transformation.
- 5. Cloud Migration
Cloud migration involves moving either data or an entire application and its data to the cloud. This can include data migration to a data warehouse or data lake. The move can be from an on-premise system to the cloud or from one cloud to another.
Planning a Data Migration
To avoid costly downtime and compatibility issues as well as corrupted, lost, and misplaced files, take time to plan your data migration. Key steps include:
Determine the Size and Scope of the Project
- Define the scope of the data migration plan
- Engage key stakeholders
- Ensure that the team has the knowledge and skills necessary to accomplish the project, or engage outside experts
- Create a realistic budget
- Conduct advanced analysis of the source and target systems
- Identify resources that are required during the data migration
- Develop a timeline that includes flexibility to accommodate the unexpected
- Determine if the data migration will interfere with normal operations
Data Analysis and Preparation
- Review the format of the data in the source system and the format of data in the target system
- Clean data as needed to ensure smooth data migration and remove or correct incomplete or incorrect data
- Determine if the data migration project will involve data integration
- Identify sensitive data and ensure that it will be protected during and after the data migration
- Assess whether the migration plans involve decommissioning an application or platform
- Remove any data that is no longer needed
- Back up all data
Define Architecture and Design Requirements
- Analyze business requirements and dependencies
- Specify what data will be moved
- Identify the tools and resources needed
- Establish security standards and data quality controls
- Define the method for transitioning quality controls to the new system
- Determine whether there will be a parallel run or a zero-downtime migration
- Develop and test different migration scenarios
- Specify the testing process
- Codify a formal data migration plan
Execute the Data Migration Plan
- Confirm that there are no connectivity problems with source and target systems
- Ensure the right system permissions are applied
- Transform data into the proper format for transfer
- Extract all data that is migrating from the source system to the target
- Load data into the target system
- Monitor data migration during the process to identify and resolve issues quickly
Migration Follow-Up and Validation
- Review the audit trails and logs to ensure that required data was transferred, if there are correct values in the destination tables, and if there was any data loss
- Test final system
- Decommission old systems
Follow-up and Maintenance of Data Migration Plan
- Monitor the data quality of the new system
- Manage ongoing improvements
- Schedule regular maintenance
Data Migration vs. Data Conversion vs. Data Integration
A number of processes are associated with data migration. Several of these are part of the actual data migration process, and others are adjacent processes. However, all are interrelated and impact data migration, including:
- Data cleansing
- Data conversion
- Data integration
- Data profiling
- Data validation
- Data quality assurance process
Data Migration vs. Data Conversion
The terms data conversion and data migration are often confusing and used interchangeably. However, there are distinct differences between the two and their roles in overall data management strategies.
When considering a data migration project, if the original and new systems have identical fields, the migration merges the two systems’ data. However, this is rarely the case.
Data conversion is usually required as a precursor to data migration.
Before the frequently complicated data migration efforts of transferring data between silos, formats, or systems can occur, data conversion is necessary to prepare the data for a smooth transition.
The data conversion process converts data from one format to another for accessibility and interoperability and ensures uniformity between the datasets. The data conversion process includes extracting data from the source, transforming it, and loading it into the new system.
The transformation is the critical part of this. Not only does data have to be in a common format, but fields must also match. For instance, if the source system uses four fields to store an address, but the new system only has three, data conversion is required to reorganize the source data into three fields.
Data Migration vs. Data Integration
While data migration and data integration are related, they are two fundamentally different processes. The similarities between data migration and data integration end with data transfers.
At the highest level, a key difference between data migration and data integration is that data migration is a one-time effort when implementing a new application, whereas data integration is an ongoing process of data transference between applications and systems in real-time or near real-time.
Unlike data migration, which handles internal information, data integration uses technical and business processes to combine disparate data from different internal and external sources to create a unified dataset for analysis. To do this, data integration enables connectivity between different systems and ensures access to the content necessary for accurate analysis.
Data Migration and the Cloud
Data migration and the cloud are often coupled with a move of data, workloads, IT resources, and applications from an on-premise environment to a cloud infrastructure, or from one cloud to another.
Data migration to the cloud often involves consolidating on-premises data warehouses in the cloud or building new cloud data warehouses and / or data lakes. To ensure data integrity, it is important to take time to prepare.
Taking time to undergo a data migration to the cloud is well worth the effort. Among the many benefits of cloud data migration are:
- Ability to ingest more types and larger amounts of data in less time
- Accelerated time to value
- Cost savings
- Easier data backup
- Flexible, sustainable, and agile data foundation
- Improved operational efficiency
- Modernized infrastructure
- Reduced IT costs
- Scalability to address changing requirements
- Streamlined data analytics with cloud processing power
- Unified infrastructure
Data Migration Results for the Effort
There are a number of reasons why data migration projects are undertaken. Many organizations choose data migration as existing platforms and legacy applications age.
The catalysts are often a need for increased performance, more and flexible storage, reliability, and security. A desire for an overall digital transformation is also a reason that organizations undertake data migrations.
Regardless of the motivation for it, data migration provides tangible benefits, including:
- Decreased service disruptions
- Improved ability to scale resources
- Increased productivity and efficiency
- Reduced storage costs
- Upgraded applications and services
Egnyte has experts ready to answer your questions. For more than a decade, Egnyte has helped more than 16,000 customers with millions of customers worldwide.
Last Updated: 6th January, 2022
