Šmonland

I recently had the opportunity to migrate a five-year old project from Subversion (SVN) to Visual Studio Online (VSOnline). Actually, at the start of the project we were really migrating to the Team Foundation Service, but then the service went live as Visual Studio Online in the middle of actually migrating content! Despite this, the migration barely skipped a beat.

Why Visual Studio Online?

While Visual Studio Online is still somewhat in its infancy, it is certainly coming along and the TFS Preview has been available for quite some time. The build and test components have also now been officially released which help complete the picture. In our client’s case however, their main reason for moving to Visual Studio Online was to simplify task and source control management among geographically dispersed teams. They had started using the Scrum template in Visual Studio Online for work item management on another project, and wanted to bring across one of their major projects to the same system.

To migrate history or not?

When moving source code to a different source control version system, you need to determine whether to migrate source history. The alternative is simply checking in the current version of your code in to the new source control system. If you decide to migrate history, how much history do you really need to migrate?

Migrating without history

Migrating without history is obviously the simplest option. Some people believe quite strongly that you don’t need to migrate source history, or perhaps you only need to migrate the history for a month or two. This might be a good option in some circumstances, but it’s important to consider things like;

• Whether you’ll have a need to compare files with previous versions?
• Do you have features that you discarded in the past, and may need to resurrect in the future?
• Whether you want to keep the old version control system around, just in case?
• If you do keep the old system around, what are the costs of maintaining it? Granted, it should be cheaper and simpler running it as a read-only system as opposed to a production one.

For example, if you don’t migrate history and you want to compare a file with a previous version from before the migration, then you’re stuck manually getting the versions you require and comparing them. Proponents of this method argue that as code evolves, it is rare that you’ll need to look more than a few weeks or months in to the past. For those times that it you do need it, the old source control system can suffice.

I’ve seen a similar situation when a client migrated to a new accounting system. They decided not to migrate historical data, as it was not frequently used. The problem though, was that the old accounting system ran on an old SUN SPARCstation 20 (yes workstation, not even server). Occasionally, the staff from the accounting department would access this system from an old Windows 95 machine that was hidden away in a room with a network port patched through to this ‘server’. The client did maintain its backup schedule, and the tapes it used were kept in the standard daily backup cycle along with all other media.

Inevitably, the system failed one day. Actually, it was one of the disks that kicked the bucket. Given the fact that the system wasn’t frequently used, there wasn’t a great sense of urgency to get it back up and running again. In the end though, I was the lucky one who had to diagnose the problem, and then figure out how to repartition the disks and restore from tape. Why did I have to figure this out? Well, because it was so well documented of course!

That little anecdote is obviously a rather extreme example. Running an old version control system on the x86 platform would certainly be less risky. At some point however, you are likely to run in to this issue, or face having to discard the history.

Migrating with history

Migrating with history gives you all of the usual benefits of using source control. It also means that you need to figure out how to migrate the data.

Out client decided they wanted to migrate with history for a few reasons. The main one, was that they were in the process of radically changing the development process of their application, and wanted to perform analysis of the historical code. This analysis would give them a feel for the quality of code over time, and also form a baseline to compare with code quality in the future.

Migration tools

If you are truly masochistic, you could always get each revision and manually and check it in to Visual Studio Online :). I suppose this might be feasible if you have a really small number of revisions, but if you only have that many why bother migrating to begin with. I investigated a few migration tools to perform the migration for our client.

First, I looked at two open source tools; TFS Integration Platform and SvnBridge. The TFS Integration Platform initially looked promising, especially given that one of my colleagues at SSW had used it before. While the tool does have SVN support, it is not in the latest official release. Because of this I tested with the latest version of the TFS Integration Platform from Codeplex. We timeboxed a task to investigate the tool, but unfortunately I couldn’t easily get the SVN support to work in the allocated timeframe.

The other open source tool, SvnBridge is used by Codeplex to provide an SVN interface to its projects. I thought that it might be possible to use this to connect to Visual Studio Online. The idea was then to use SVN dump and load to migrate to Visual Studio Online through SvnBridge. As with TFS Integration Platform, I ran out of time trying to get SvnBridge to connect to Visual Studio Online, and suspect that it may not be supported due to the way that SvnBridge connects to TFS.

After the difficulty we faced with the open source tools, we decided to look at the commercial tool Timely Migration. Timely Migration supports migrating from SVN to TFS. It doesn’t explicitly say anything about Visual Studio Online on their website, but a quick test revealed that the tool did in fact work with Visual Studio Online. I contacted the guys at Timely Migration, and they happily confirmed support for Visual Studio Online, adding that they had recently completed a migration to the online service for another client. The cost of the Timely Migration SubversionToTFS module at the time we needed it was only US$1995, which I think is quite reasonable. Based on this testing and great response from the vendor, we went ahead with Timely Migration.

Using Timely Migration

The Timely Migration tool is fairly straight forward to use, and the documentation easy to follow. You must configure it to connect to your source and destination repositories, a SQL instance for staging data, then map folders from the old to the new repository, and that’s about it. There are also some advanced settings which allow you to tweak how it moves the data across.

I won’t go through all of the settings in the tool, however it is worth touching on a few which we used. For example, by default the check-in comment of all migrated revisions from SVN is in the format {$(DateTime) r$(Name)} $(Comment) (Migrated by SVNToTFS). This can easily be changed in the settings tab.

Timely Migration connecting to Visual Studio Online

The first thing that I tested was connecting to Visual Studio Online. I did this via the Destination tab in Timely Migration. From here, I clicked on the ellipses button to bring up the standard ‘Connect to Team Project Collection’ window.

Figure 1 Destination tab in Timely Migration

Figure 2 Standard ‘Connect to Team Project Collection’ window from Visual Studio

From here, it’s a case of adding your Visual Studio Online URL to the list of Team Foundation Servers, just like in Visual Studio.

Figure 3 Add/Remove Team Foundation Server dialogue box

Figure 4 Add Team Foundation Server dialogue box, using the URL of my Visual Studio Online subscription

Because I added a Visual Studio Online URL, it to sign in using my Microsoft Account:

Figure 5 Dialogue box: Sign in to Team Foundation Server using Microsoft Account

Once signed in, you can select your Team Project Collection and connect. You can connect to your DefaultCollection Team Project Collection. For those of you not already aware, you are currently limited to the DefaultCollection Team Project Collection in Visual Studio Online.

Figure 6 Dialogue box: Connected to Visual Studio Online

Back-dating time stamps in the destination repository

One thing that our client was particularly looking for, was to have all of the original time stamps from SVN imported correctly to Visual Studio Online. They were interested in this because it would make the historical analysis that I talked about at the beginning of this post easier to implement. The Timely Migration tool does in fact support doing this, using the BackDateTimestamps option:

Figure 7 Settings tab in Timely Migration

I did some testing with on my own Visual Studio Online subscription with this setting set to true. The setting did work, however I noticed a side effect that another Team Project had its dates affected. After testing some more, I got an error which prevented any further migrations

Here are the steps I used to test this process.

1. First, I deleted all team projects in my Visual Studio Online subscription
2. I then created a new project called ‘Existing Project’
3. I checked in a dummy console application to this newly created ‘Existing Project’:
   

   Figure 8 Initial check in of dummy project, showing check in time of 2 minutes ago
   

4. I created a new team project with the name ‘Anvil’
5. I migrated the first 500 revisions using Timely Migration, with the ‘BackDateTimestamps’ option set to ‘true’
   

   Figure 9 The first few items of the Anvil project migrated, with a modified date of 14/10/2008
   

6. After the migration started, the dummy project showed different ‘last modified’ dates, which corresponded to the first check in date of the migrated Anvil project:
   

   Figure 10 Dummy project showing last modified date of 14/10/2008
   

7. While leaving both existing projects alone, I created a new project called “New Project”. This project correctly showed the last change value as a few minutes ago.
8. I then started another migration, this time with revisions 501 to 550. This failed with an abort message and the error:

   Migration error of TF203096: Your attempt to reset the check-in dates in the team project collection failed because one or more of the team projects that it contains are not empty.

Because our client had existing Team Projects in their Visual Studio Online subscription, I had to migrate the source without back-dating time stamps. Even though this was the case, at least the time stamp information came across in the comments. Although not ideal, it does mean that it will still be possible get the original time stamp by parsing the comments text.

Migration performance

Overall the migration performance was quite reasonable. It looked like our 8GB SVN repository would be migrated in around less than 70 hours, on a 50Mb down / 4Mb up link. After a couple of days however, the speed started to significantly slow down. Individual revisions were taking many minutes to complete. Alongside this, I noticed the disk space of the migration machine filling up very quickly. This was because the log files from Timely Migration were growing to many gigabytes in size.

I got in touch with John from Timely Development, who after having a look at the logs came back with the following:

So the issue does appear somewhat related to the hosted TFS / Visual Studio Online server.

Back with TFS 2008, asking TFS to apply a label recursively would sometimes mess up and label files that were deleted, so when you retrieved the label it would return the last version of the deleted file before the delete. Because of this the tool had to correct the label problems.

When TFS 2010 came out label handling changed so that wasn’t an issue anymore, but the TFS API didn’t provide a way to check server versions to easily disable the label validation. So the check was to look at the URL scheme to see if it had a collection in it, which doesn’t work on the visualstudio.com URLs.

So aside from the overhead of the server communication, I’m not sure this should have any affect. To speed things up though, if you exit the configuration UI and manually add the following configuration setting in the XML file near the “IncludeLabels” setting, it will skip the check when the session is restarted.

<Setting name=”ValidateRecursiveLabels” value=”false” />

After changing this setting as he suggested, the migration immediately picked up and completed just over a day later. The log files also stopped growing.

Conclusion

Overall the process went fairly smoothly, and we achieved our main goal to migrate our client’s source history so that they could start working from Visual Studio Online. The guys at Timely Development were very responsive and helpful both before and after our client purchased the Timely Migration tool. Their help gave us confidence in clarifying Visual Studio Online support, behaviour of back-dating of time stamps, and helped us get over the speed issue we encountered. It’s also worth noting that their SVN tags were correctly migrated to TFS labels.

As well as the migration, I also attempted to reconfigure their TeamCity build server to work with Visual Studio Online. This turned out to be a bit more of a challenge than initially anticipated, as the required version of Team Explorer wouldn’t run on their old Windows Server 2003 build server. I ended up building them a new build server with Windows Server 2012, and upgraded the TeamCity configuration on this new server.

Have you had any similar migration experiences? Or has this helped you through your own migration? Please let me know in the comments below!