Why is Data Migration so difficult?

“It’s just like learning to ride a bike – if you don’t use stabilisers and / or have someone show you how…you are going to get cuts and bruises?”

Author: Fay MacDonald 27th October 2011

What is Data Migration?

Data Migration is the transfer or copying of data from one IT system or multiple IT systems to another IT system. It typically involves:

  • the use of a standard set of scripts which the supplier of the receiving system will provide;
  • a standard tool (hardware and software) provided by the supplier of the receiving system;
  • a technical engineer with specific experience of data migration projects and scripting;
  • involvement of an operational team of people from the organisation who is changing the systems to check and test the data using templated and standardised test scripts;
  • a data cleansing team;
  • lots of planning, managing and testing – best overseen by a project manager experienced in data migrations;
  • gobbling up a fairly large pot of the project budget and then the Oliver Twist moment “please Sir, may I have some more”;
  • tough decisions on whether it is absolutely necessary to migrate all the data – it may be a high financial cost for a poor return!

What are the typical pitfalls?

One of the biggest mistakes that is made for Data Migration projects is insufficient preparation up-front during procurement and planning. There is little or no time and effort spent during this period to prepare a pilot activity to test the water on data completeness and compatability of data structures between the 2 systems which will substantiate and inform the outline project plan and budget.

Many projects kick-off straight from a procurement process with a contract signature between the IT systems supplier and the procuring organisation. The contract very often will state that the data “can be migrated” and that “the IT system is capable of receiving data”, with a usual caveat of “a standard format, e.g. HTML”. This all sounds great so far.

The trouble is that these kind of statements are so high-level that in practice,it will not actually predict the time and costs involved for either party and boy oh boy there is usually a big shock followed by many gasps and expletives uttered when the impact in terms of costs and time required are known.

What is typically found on floundering and failing projects which include data migration is that they are in mid-flight of the deployment project and that:

  • IN-EXPERIENCE OF THE PROCUREMENT TEAM: the people involved in the procurement from the client/ procuring organisation  and the sales team from the supplier have never undergone a project with data migration;
  • IN-EXPERIENCE OF THE SUPPLIER TEAM: the IT supplier is typically not in the business of specialist data migration work, other than to “receive” the data;
  • GAPS / SILENCE OF THE CONTRACT: the contract wording does not actually obligate the supplier to re-structure data, to perform any pilot work or to take the decisions. There are no explicit clauses and responsibilities therefore neither party has envisaged that they will be commercially implicated to deliver this middle-ground piece of work between the 2 systems: the old and the new.

This kind of work very often falls through the crack of joint responsibility ….which means that often neither party to the contract takes true ownership. Each party naively assumes that the other knows the implications and scope. HOW WRONG THIS ASSUMPTION IS! The cost and overrun is found out mid-flight of the project. Toys are thrown out of the pram by both parties…the project delivers late and costs more for both parties. A bad experience is had by all and war wounds abound.

The analogy which comes to mind is of people emigrating / immigrating. There are so many people of many different nationalities who have migrated to other countries over the last couple of hundred years. Modern day requirements for immigration to most countries have a checklist of requirements which an applicant must meet in order to gain entry, be granted a visa to live and work and citizenship:

Border crossing - checking the entry requirements

  • border control and visa regulations are in place which will require an application from the person which will have a standard set of questions and requirements in terms of ethnic origin, language, skills etc.;
  • the requirements for entry to the “new” country are essentially similar to the concept of the minimum data set requirements of an IT system when moving data from one system to another;
  • if the person meets the requirements they will be admitted to the country;
  • however if the person does not meet the minimum entry requirements criteria they will be refused entry. This compares to the data migration scenario where if some of the data elements were missing or mis-matched in some way, the data would end up in a failed record queue and would not be imported to the new system.
  • a decision then needs to be made for the person and for the data: what happens next…can the minimum entry criteria be achieved for the person e.g. if they need to acquire new skills or language proficiency, can they do this and re-apply. For the data, can the missing fields be completed accurately and then have a re-load of the data to the new system;

PAY THE SPECIALISTS FOR ADVICE EARLY OR BREEZE IN AND RISK PAYING SUBSTANTIALLY MORE LATER: In these scenarios success in terms of entry may be achievable. However to do this whether it is a person acquiring new skills or data being enhanced or cleaned up will take time and money, as it will usually involve paying third parties to assist in the process.

How can the pitfalls be avoided to ensure a succesful project?

Data migration is an activity that has many complexities, however to be successful the steps to manage and control the work can be simple.

RULE # 1

Involve specialists who have experience of data migration projects as early as possible during the Project procurement.  Do not wait until the deployment starts.


Follow a structured process to assess, evaluate, plan, execute, test and sign-off on the data migration process.

There needs to be clear scoping and a “gating” or decision-making process at each gate. This will typically involve project board members and any project governance bodies as defined in the Project Initiation Document (assuming that such a document has been created at the procurement stage).

5 Simple Steps to Follow

The steps I have used on many projects and would recommend are outlined in the table below:


T w i t t e r