Many organisations are now considering a move to the cloud from legacy on-premise solutions and systems. There are several considerations as you review each application that may be migrated but underlying each application migration project is a data migration exercise that moves all your associated data from your servers in-house into the cloud service providers. Data migrations can be one of the most complex aspects of the move to the cloud and to get it right requires careful planning. This post outlines 7 key steps in this process and will help make your cloud migration successful.
1. Analyse the data
You really need to “know your data” before you look at any migration. Understanding what data is where and why it’s there (application data, reporting data, etc) is important in ensuring your business will continue to run as it does today post the migration. Some organisations know their data a lot better than others – and this can be because they don’t have a lot of volume, or because they have put in the effort to understand and catalogue the data in the organisation. Tools such as Alex Solutions and Microsoft Purvue, alongside other data cataloguing solutions, can help in this regard. Understanding how each data asset is used and its lineage, or how it is connected to each application that uses it, will help ensure the migration is successful and will also allow you to determine which data assets can be decommissioned as part of the migration process.
Once the data has been analysed you can also review the quality of the data and determine what sort of clean-up process may be required prior to or during the migration. No matter how well you manage your data you can be sure you will have issues with quality as very few organisations have business-wide data reviews to ensure data quality remains at a certain standard, even if those standards are well defined and documented.
2. Plan and Design the migration process
Planning will help ensure business continuity during the process. If you have large amounts of data, you will most likely want to “chunk” the migration process down as moving large amounts of data will take time. Your previous step will have informed which chunks of data must move together and this will be an important input into the planning process. Ensuring all business applications work as expected post-migration requires careful planning and understanding of each of your applications and the data used.
If you are going for a “big bang” migration rather than chunking your data down, then testing timings will be important to make sure you have enough scheduled downtime (assuming your applications need downtime – most will) to transfer the data to the new location. This can raise pressure on the process as the business will probably need to operate without one or more of their applications online during the process. Test runs on a development server will help but these test runs don’t always give accurate timings as production servers and production data can often be very different to what happens is stored in our test and development systems.
Whether you decide to migrate all your data together, or small amounts at a time (the latter is considered the least risky), careful planning commensurate with the size of your organisation and the amount of data to be migrated is highly recommended. What is also recommended is that you very carefully analyse your data before your start, so that you know in what state it is in (from a quality perspective) and exactly where it all is – for example, I once worked with a client where all their CSM’s were storing the phone number in the 2nd address line because their systems were so old they didn’t allow for the storage of a mobile number.
3. Choose the right tool to assist and make sure the people are in place to assist.
The data migration process is essentially an ETL process where you will extract the data from your existing data store, transform it (if required) to suit the new environment, and then load it into the new environment.
A number of the cloud vendors have tools that you can use to assist with the migration process, or there are many open source solutions that can help. Ensuring whatever tool you choose can do the job by thoroughly testing, or asking an independent third party, will help make sure your migration happens smoothly and as efficiently as possible.
4. Make sure backups are in place and current
Data migrations, like any complex technical process, can go wrong. There is added risk with data migrations as they often need to run over many days or sometimes even months. I remember doing a SharePoint migration for a large enterprise where the migration was run over many months and this added complexity to the whole process as well as careful consideration of the business impact, which was significant.
5. Test, Test, Test
Oh, but it’s only a “lift and shift” they said. These are famous last words from those who don’t always understand the complexity of moving significant amounts of data from one system (often on-premise) to another (usually in the cloud) – particularly if there is any reformatting of that data along the way.
Testing with small chunks is critical. It’s ideal to test with real data if possible as you will get a good idea of whether or not there are going to be issues with the quality of the data being moved. If our transformation programs all assume a certain value in a certain field and then we find that all our front-line staff have been using that field for something completely different, then it can cause issues with the migration with either timing or worse, with a process that fails part way through.
6. Execute the plan Are the people in place? Check Are the migration tools in place? Check. Are the migration processes in place? Check.
Backups are done and current. Check. Source systems and target systems are all available and sized accordingly? Check. The business has been informed and knows exactly how long they will be without their systems and what they will need to do when those systems come back online. Check.
Post-migration audit procedures are in place? Check.
Ok, let’s go (note the above is not necessarily a comprehensive checklist for your go/no go decision!).
Time to begin the migration. Because the timing of data migrations is so important, the initial hours should be monitored closely to ensure the migration is progressing as expected. If things are going much more slowly than anticipated it may be that one or more of the assumptions made, or the tools being used, is not performing as expected – and the 2-day or 2-week window you have made available may not be enough time. You can usually tell this within hours and if necessary, it is better to call the migration off and understand why it was not performing as expected than to get 80% of the way there and have the business systems unavailable for an additional week – with the business now suffering and pressure increasing on the technical team. This is often where mistakes can happen, or worse still, the business can be adversely impacted by the unavailability of a key or critical system.
7. Follow up and ensure data quality Even with proper testing, there may still be some issues with the data or the systems you are migrating. Having your application experts and super users from the business available to test all the systems before they are made available to the rest of the organisation will ensure small issues don’t become larger ones. By picking up any “niggles” with the process they can be addressed before all users are allowed back in to use the applications. The data migration process can be long, and it can be complex, but following the above process and being thorough should mean a successful migration to your new environment. If done right, it should also mean cleaner, better-quality data and easier to use applications, both from a user perspective and a technical perspective. As migrations to the cloud increase, refining these processes and making them repeatable becomes more important. Sometimes it is best to have someone help define those processes with you upfront – someone that’s has been through this process before and knows the pitfalls. If you don’t have those people in house, then it can be worthwhile hiring a consultancy to assist. It will be money well spent when your first migration goes as smoothly as your final migration.