In the Life Sciences industry a structured, but lean, approach to data migration is vital for success. Such an approach not only ensures data integrity and consistency, but also minimizes the data migration impact on day-by-day business operations.
On May 14, Arithmos hosted a complimentary webinar on performing a successful data migration in Life Sciences. The webinar described the main risks and challenges of the data migration, ways to overcome them, and presented a case study of data migration in pharmacovigilance.
The webinar was conducted by Alessandro Longoni, a Senior Business Analyst & Project Manager of Arithmos. He has been working in the field of Life Sciences technology for 13 years and has particular expertise in pharmacovigilance and electronic data management.
Continue reading to discover the questions from the Q&A session from the webinar.
During a pharmacovigilance migration, should only the last version of the cases be migrated, or should we migrate all the versions?
It depends on the legacy and the target system. Some classical safety systems save each version of each case as an independent case.
If you use an ETL tool, to migrate to a new system, that does not operate according to this logic, we would advise to migrate only the last version of the case in order to avoid the duplicate cases. It is also advisable to migrate the case history (case workflow) if possible.
However, in all the cases, it is technically possible to migrate all the versions of the case if you find the right workaround. For example using the following naming format: <case_number>_<caseversion>.
The best practice in data migration is adopting the E2B XML approach – regardless of the number of case versions, only the last version is included in the legacy system.
What steps can be omitted in semi-automated data migration compared to the automated?
Data migration, regardless of the approach used (ETL, XML import, re-entry), must be included in a validation exercise.
The main steps and deliverables are the same regardless of the approach. However, the documents and activities in semi-automated and automated data migration projects require different accuracy and effort:
- Data Migration Plan: it should include the strategy of the migration project and the main rules for data migration. This means that it covers the mapping needs for re-entry, and in case of the ETL tool, a data mapping document is highly suggested.
In case of the XML migration the format depends on the number of cases and the complexity (due to the difference between the configurations of target and legacy system). For simple migrations/low number of cases, the data migration plan could be sufficient. For complex migrations/high number of cases, we suggest a dedicated data mapping document
- Data Mapping Specifications: see above
- Data Migration Validation: it is a protocol that must be executed in the validation environment to verify the migration process. It is more extended in the case of ETL, but we suggest to use it regardless of data migration approaches.
- Data Migration Production: it consists of the execution of the protocol, described above, that is conducted to migrate data in production. In case of an ETL approach, this phase will require 2-5 days for technical activities plus 1-2 days for business data verification (production downtime period). In case of an XML or re-entry process, this phase will take up to several months of business work (no production downtime period).
- Data Migration Report
When you access the source data in read-only mode, is it possible to also access the files attached to the ICSR such as publication (in PDF format) or ACK from Regulatory Authorities (in XML format)? / How can we migrate attachments from one system to another? Should the client send the attachments separately from the XML files if the migration is performed by import/export migration?
Yes, it is possible. Of course, this possibility depends heavily on the structure of the legacy system database. Sometimes, technology constraints don’t allow you to extract some particular information, such as attachments or audit trail.
If the legacy system stores the attachments in the database with proper references to the relevant case, and if the target system supports the case attachment functionality, the best idea would be to perform the integration using the ETL tool.
If I have an E2B XML file as source of a case recorded in an Excel database, can I use it as a source for the migration into the new database in order to avoid doing a migration manually from the old database?
The standard approach to data migration is migrating data from the legacy to the target system and not from the source to the target. Nevertheless, a different approach can be chosen if it can grant a better data quality (like in the question) and efficiency. This approach must be described and justified in the Data Migration Plan. Also, we advise to carry out a final verification to detect every discrepancy between migrated data and data stored in the legacy system.
How can you calculate the number of the records from the source and target systems that need to be validated?
An ETL tool can reduce the need for the data verification, especially because a migration script works on specific fields independently from the number of records to be migrated. For example, if my script moves correctly the laboratory data, the number of tests executed by a specific patient is not relevant. This ensures the quality of the migrated data.
However, a minimum quality check of the fields has to be performed, and it should be based on the recommendations of your CSV manager and the risk assessment.
A numeric check of the records moved, (100%), is always required to ensure that there were no exceptions during the script run.