Microsoft Dynamics CRM 4.0 Data Migration Performance

Koushik Bhattacharjee wrote on June 02, 2008 8:15 AM

---

This blog post is intended to provide necessary information regarding performance of the Data Migration feature. I will provide some details on the tests and results on a specific hardware environment and some guidance on the deployment side when performing a high volume data migration.

Hardware Configuration:

I have used this hardware set for the data migration tests covered in this blog.

• Domain Controller : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
• Application Server : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
• SQL Server : Dell Power Edge 2850, 2 Proc Xeon 3.6 GHz, 2 MB L2 Cache, 800 MHz FSB, 4 GB DDR2 RAM
• DMM Machine : HP Compaq dc 7700 SFF Intel Core 2 Duo 2 GHz, 2 MB L2 Cache, 800 MHz FSB, 2 GB DDR2 RAM

Software Configuration: The domain controller, application server and the SQL server were running Windows 2003 Enterprise server sp2. The SQL server was configured to run an instance of SQL server 2005 Enterprise edition sp2. The DMM machine was prepared with Windows XP professional edition sp2 and was running an instance of SQL Server 2005 Standard edition sp2. All the latest hotfixes and updates were applied in the machines.

Workload: The Migration scenario is valid for a fresh installation. The system was having one organization with 200 users. One user (with system administrator role) was dedicated for data migration. There was no static data load or run time load of users on the test system when migration was in progress.

System Settings: No workflow was defined during the course of migration. Duplicate detection was turned off.

Scenarios:

Based on different size and need of the organization, we came up with the following three migration scenarios.

1. Simple: An organization migrates Leads, Contacts, Campaigns and similar entities without any relationship. This is the simplest form of data and does not involve any complex mapping or lookup resolution. DMM intelligently migrates this type of independent data in a parallel fashion and achieves very high throughput.
2. Moderate: An organization migrates records with picklist and lookup fields defined. But there is no complex transformation defined.
3. Complex: An organization using some other CRM, migrates to Microsoft CRM. There are multiple entities with rich interdepending relationships and large number of records. The used map involves picklist, lookup and complex transformations.
4. All these three scenarios are unique in nature and the migration time can be very different with these situations. I am listing the scenarios and the end to end throughput (records per second) that I got while executing these scenarios.

Scenario Complexity	Data Size	Migration Time	End to end throughput (records per second)
Simple (Without any picklist, lookup or complex transformations)	5 million	20 hr	70
Moderate (picklist and lookup transformation but no complex transformation)	1 million	9.2 hr	30
Complex (14 CRM entities with picklist, lookup and complex transformation)	2.5 million	37 hr	19

Let me elaborate more on the complex migration scenario. The migration data was very high volume and spanning over 14 CRM entities with SF like data. The entity wide record distribution is provided in the below table.

Entity	Count	Entity	Count
Account	100000	Opportunity	500000
Contact	200000	OpportunityProduct	500000
Lead	500000	Product	10000
Contract	200000	PriceLevel	10000
Case	100000	ProductPriceLevel	10000
Event	100000	Solution	10000
Task	200000	Campaign	10000

The file sizes for some of the entities were more than 32 MB and hence I had to split them and adjust the out of box SF map. The resultant map was mostly same as the original map, only I need to duplicate the mapping for the entities where I had split the data files.

The above tests show that DMM can handle data of different complexity and can migrate really high volume of data. The migration time and end to end throughput actually varies a lot depending on the complexity of your data. But these results will give you a starting point if you wish to know how much time the migration may take in your deployment.

Tips for high volume Data Migration:

Before you do a migration involving huge number of records, I suggest you to do the following steps.

1. Deploy your CRM server according to your business need and targeted capacity. DMM should be installed in a 2 proc 32 bit machine with at least 4 GB RAM.
2. First do a test migration with very lees records but the same map and same type of data. This will help you to know whether the map is appropriate and you can undo the test migration, by using “Delete Data” option in DMM.
3. If you do not want to automatically customize the CRM server during migration, then uncheck that option during migration. The repercussion is that custom attributes of picklist types will not be populated automatically.
4. Do not migrate data using a virtual machine. Use a physical high end machine to achieve faster migration.
5. Follow the below mentioned optimization techniques, wherever possible.

Optimization Techniques:

1. All the machines in the system must be in the same time zone and use same regional settings.
2. In the Internet Options -> Connections -> LAN Settings, set the physically nearest proxy server and set ‘Bypass proxy server for local addresses’. Do not use the option ‘Automatic Configuration’ as it involves in IL code generation (observed for .NET 2.0 SOAP classes).
3. For best results, make sure that the DMM and CRM server are on a fast LAN connection with no TCP routing hops or HTTP proxies in between.
4. Dynamics CRM 4.0 application has been launched at least once before testing.
5. Make sure that tracing is off in both server and DMM. Tracing might drastically slowdown the application.
6. For best results, CRM server, SQL server and DMM should be in separate boxes.
7. If user has a license for SQL Server 2005, then it is advisable to have DMM database in that SQL server and not in SQL Server 2005 Express edition. SQL Express is not advisable if you intend to do a high volume data migration.
8. Follow the optimization techniques mentioned in the whitepaper Optimizing and Maintaining Microsoft Dynamics CRM 4.0.

Conclusion:

The above mentioned results outline certain aspects of Data Migration performance for Dynamics CRM 4.0. The combination of fast algorithms used in Microsoft Dynamics CRM 4.0 Data Migration feature, along with the new generation Intel Xeon servers results in the high throughput for complex Asynchronous Data Migration jobs.

When reading this blog, user should keep in mind the following points.

• The above results are obtained when only one user is performing Data Migration. The results may not be same if multiple users are performing data migration or other operations are being executed concurrently.
• This test was done in an on premise CRM deployment and the results may not be same for CRM online.
• There is no workflow or plug-in defined during the course of migration.

Any deployment in a similar environment may not provide the exact same numbers, but the provided numbers indicate the pattern of performance numbers for Data Migration. This can be useful in initial planning for migrating your data into CRM using Data Migration Manager.

Koushik Bhattacharjee