Realize business value by placing valuable data together
Moving data from one system tot the other is also called data migration. This is a challenging exercise that heavily relies on data definitions, business rules and data quality.
LeanData provides processes, methods and solutions to facilitate the movement of data in a controlled, reliable and efficient manner.
It starts with getting the business and IT objectives straight. Just to be clear, data migration is not about just moving the data. It is about considering the business processes, the purpose of those processes and the corresponding entire data lifecycle from creation, use and cleanup & removal.
Data Migration is about ending (the processes supported by) the old system, not just moving data into a new system
LeanData fundamentals in managing data migration
As data is the result of business processes, we can evaluate the business processes effectiveness and efficiency by looking at the data produced. Every time oddities are observed, you should review the business process steps and improve them to get them in line.
The structured approach LEAN describes to improve processes can also be applied to data processes. Data Migration starts with applying Kaizen 5S methodology:
This requires us to understand where we are and where we are going. Thus first profile the data in the current system to understand what it is. There we will quickly see there are data ‘sections’ that we do not need to move, data that is good as it is and data that needs to be transformed to fit the target system.
Now by sorting the datasets we want to migrate we quickly see patterns.
- Data fields that are not filled
- Data that looks like other data
- Data that is not unique
- Data with unusual or unexpected values
With previous steps we gain an understanding of our data and what we actually want. We can correct and clean up the data before we actually move the data into the new system.
While cleaning up the data it is also important to reduce its complexity. This is usually caused by workarounds of our processes and results in unmanageable variation of the data produced. Often things are misunderstood and the system used in the wrong way.
This we need to apply in the new system in order to maintain the data quality required. By documenting the previous steps we will be able to continuously improve the business processes.
- First thing to understand is the difference between the current and new business processes that need to be supported for which a new system is to be implemented.
- When the goals are clear, a fit gap analysis can be made that will guide the required data transformations based on the documented definitions.
- We can then create the Logical Data Model for the new system and understand the data requirements.
- Now profile the data from the source system and identify the deviations with the LeanData Deviation Discovery solution. The key is to work together with the business who knows and understands the data itself.
- This will lead to clean up and data transformation rules.
- Then map the data between the source and the target system.
- Start cleaning the data and transform it to the data structure of the target system.
- Move the data to the target system and closely monitor the process for exceptions.
- Test and validate the processes and the data quality of the target system.
- Complete the documentation with decisions, definitions, business rules and data quality rules in order to support reliability and further growth on the target system and maintain data quality to stay clean.
The primary solutions to facilitate data migration can be found here:data deviation discovery