P3.NET

Migrating from TFS to VSTS, Part 5 – Build Components

13 April 2018
Categories: TFS

In the final post on our migration from TFS to VSTS we will focus on the build components of TFS. This includes build definitions and task groups.

Part 1 – TfsMigration Tool

Part 2 – Migrating from TFVC to Git

Part 3 – Migrating Areas and Iterations

Part 4 – Migrating Work Items

Part 5 – Migrating Build Components

In our TFS environment we used build definitions using Build vNext to automate our builds. We use custom build tasks for our builds and we use task groups to consolidate the common build steps needed by all our builds. This is certainly not all the build components of TFS. Things that were not migrated are covered at the end of the post.

Migration Process for Build Components

The BuildManagement processor is responsible for migrating build definitions and templates. The settings for build components are contained under buildManagement.

"buildManagement": {
   "copyTemplates": true,
   "overwrite": true,
   "targetAgentQueue": "Hosted",
   "excludeDefinitions: [
      "TestDefinition
   ],
   "taskGroups": [
      {
         "sourceGroupId": "Guid-1",
         "targetGroupId": "Guid-2"
      }
   ]
}

Note: Before the migration tool can migrate anything that relies on custom build tasks they must be installed on the VSTS account. Ensure any appropriate extensions are installed first otherwise migration will fail.

Migrating Task Groups

If your build uses task groups to simplify the build definitions then they must be migrated first. Unfortunately the REST API (at the time of this writing) does not provide support for this. To migrate a task group you must export the task group from TFS and then import it into VSTS. Support for this was added summer of 2017 to the UI.

Unfortunately, because the API does not support export/import of task groups, the group’s ID will change. The group ID is displayed in the URL when editing the task group and is a GUID. When a task group is used in a build definition the group’s ID is stored. The ID has to be fixed up during the migration. The taskGroups setting array is available for this.

  1. Add an entry for each task group that needs to be exported.
  2. Set the sourceGroupId to the group’s ID in TFS.
  3. Export the task group using the UI.
  4. Import the task group into VSTS.
  5. Set the targetGroupID to the group’s ID in VSTS.

Migrating Build Templates

If you have custom build templates in TFS then you may want to migrate them. If you do not know what a build template is then create a new build definition. When you do you will see a list of pre-defined templates that you can use as a starter. That is a build template.

If the copyTemplates setting is true then custom build templates will be migrated. The processor starts by getting all the build templates defined on the TFS server. This is done using the GetTemplatesAsync off of BuildHttpClient. The same is done on the VSTS server. The CanDelete property is used to identify custom templates. Only custom templates will be migrated.

For each source template the processor sess if it is already in VSTS. If it is then it is skipped otherwise the SaveTemplateAsync method is called to save the template. No changes need to be made to migrate it.

Migrating Build Definitions

The processor starts by getting all the build definitions from TFS using the GetFullDefinitionsAsync method on BuildHttpClient. For each build definition the processor then determines if the definition is on the exclude list in the settings file. If so then it is skipped.

The processor then checks to see if the build definition already exists in VSTS. At this time there is no API for getting a definition by its name. Instead the processor has to retrieve all the definitions and then use LINQ to find the specific one. It would probably have been faster to cache this but the processor does not. If the definition exists and overwrite is not set then the definition is skipped. Otherwise the definition is deleted using DeleteDefinitionAsync.

With the checks out of the way the processor can now migrate the definition. The PrepareDefinition method cleans up the definition and then the definition is added using CreateDefinitionAsync. The PrepareDefinition method clears out some definition settings that aren’t applicable (or will be set) when created in TFS. The queue is set to whatever TargetAgentQueue is set to the settings file.

TO ensure the task groups get updated properly PrepareDefinition enumerates the Steps in the definition. For each step the processor looks at the TaskDefinition.Id property. For a custom task group that will match the source group in the settings file. If found then it is replaced with the target value. When the definition is created it will be referencing the new task group.

Note: The build definition in the version of the API targeted by TfsMigrate has been replaced with a newer definintion that exposes the steps differently in preparation for YAML.

Limitations of the Build Migration

The processor for build migrations has some limitations.

  1. The processor does not know the path to the source in VSTS. It is unlikely that it is the same as the TFS server. The source path needs to be fixed up manually after the migration. As an enhancement the processor could be updated to specify the path to each source path or perhaps rely on the existing version control processor (if it is being used).
  2. Any defined triggers are not explicitly migrated. Each build definition’s options should be verified after the migration to ensure they are correct before turning the builds back on.

Migration Process for Packages

The PackageManagement processor is responsible for migrating NuGet packages. The settings for packages are contained under packageManagement.

"packageManagement": {
   "nugetCommandLine": "C:\\Program Files (x86)\NuGet\nuget.exe",

   "sourcUrl": "https://mytfs:8080/DefaultCollection",
   "sourceFeed": "SourceProject",

   "targetUrl": "https://mytfs.visualstudio.com",
   "targetFeed": "TargetFeed",
   "targetPackageSource": "LocalSourceName",

   "overwrite": false,
   "includeDelistedVersion": false, 
   "latestVersionOnly": false,

   "excludePackages": [
      "ObsoletePackage"
   ] 
}

Creating a Package HTTP Client

Package management is only partially supported by the TFS REST API at this time. The migration tool was created using version 2.0-preview.1. For the most part you can get and view package feeds and information but not add or update. This makes the processor more complicated than normal.

For working with the package files themselves the processor uses NuGet. The NuGetCommand class is a wrapper around calls to NuGet. Because NuGet doesn’t have access to the same information as VSTS you have to configure a NuGet package source for VSTS. On the source TFS side the processor can use the API. Because the NuGet command line is used that also means the processor has to store the packages on disk, at least temporarily.

On the API side there was also no HTTP client for packaging. A custom PackagingHttpClient class is defined to implement the client similar to how the existing API clients are defined. It even derives from VssHttpClientBase in order to handle the credential management. In all the client calls it is important to pay attention to whether the feed or package URL is needed. The settings file stores the feed URL. To convert to the package URL the code replaces the .feeds. value with .pkgs.. This works in the current versions of VSTS. The feed URL is needed when getting information about all packages or adding packages. The package URL is needed when working with a specific package. The client handles this silently behind the scenes.

The class defines the following members.

DelistPackageAsync delists a package. It uses the exposed API endpoint to do that. To delist a package you need the feed, package name and package version to delist. A PATCH request is sent to delit it.

DownloadPackageAsync downloads a package using the same endpoint. It requires the feed, package name and package version. The package is returned as a Stream so the client returns it back directly.

GetPackageAsync gets the metadata about a specific package. Unfortunately, whether by design or through a bug, attempting to use the API to get a package by its name does not work. It always fails. So the method gets all the packages and then searches for the correct package.

GetPackagesAsync gets all the packages from a given feed using the appropriate endpoint. It optionally provides the ability to get delisted packages and some or all the versions. Because a lot of data can be returned the results are paged. Again there is a bug where not all the versions are being returned for each package. The client gets all the packages and then calls the API again to get the versions for each package. It is expensive but more reliable.

Note: The migration tool is limited to migrating NuGet packages from TFS because that is the only functionality we needed. If you need to migrate other types of packages you will need to extend the code.

Migrating (NuGet) Packages

Before using the processor the target feed has to be set up and its name set in the settings file. The API supports doing this programmatically but the processor opts not to do this automatically.

The processor starts by creating a temporary folder to store the downloaded packages. It then queries the packages in TFS using the GetPackagesAsync method on the client. For each package the processor checks to see if it is on the exclude list. This is useful since you do not want to migrate packages that are no longer in use.

For each package to be migrated the processor then gets all the versions of the package in VSTS. The processor then enumerates the versions of the source package from oldest to newest. It is important to do this because you cannot upload an older version of a package. For each version the processor checks to see if it is delisted (using the IsListed property) and whether the version already exists in VSTS. Depending upon the results the version may be skipped or not. If the version is to be migrated then it is downloaded from TFS using the SavePackageAsync extension method. This method calls the DownloadPackageAsync method on the client and then saves the package version to the file system. Note that a single package may have multiple versions.

Once the package has been downloaded it can be uploaded to VSTS using NuGet. The PublishPackageAsync method handles that work. This method uses the standard NuGet push command to push the package in the temp folder to VSTS. Unfortunately the command line does not indicate success or failure so the method then calls GetPackageAsync to get the package. If that call fails then the push failed.

The last step is to delist the version if it was delisted in TFS. The DelistPackageAsync method is called to do that if needed.

Limitations of the Package Migration

Because the APIs are limited in usefulness at the time the tool was written there are gaps in the package migration.

  • Feeds are not automatically set up if they do not exist.
  • Custom views are not created in VSTS.
  • The view(s) associated with a package are not set.
  • As mentioned earlier, only NuGet packages are supported.

Other TFS Components

Package management completes the things that are automatically migrated from TFS to VSTS. There are some areas that could be automated but were found to be easier to set up manually.

XAML build definitions are not supported on VSTS and therefore cannot be migrated. If you have XAML builds you will need to convert them to Build vNext. YAML builds are also not supported.

Release definitions are not migrated. We do not use TFS releases so support was not added. A new processor could be defined to migrate them if needed.

Build artifacts and history are not migrated. It does not appear that VSTS supports importing either of these and therefore that information will be lost. Artifacts should be stored on a file server or similar location if they are needed after the TFS server is taken offline.

Dashboards and the widgets on them are not migrated. This may be a little or a lot of work. Because you are moving servers some extensions may no longer be available or queries may not be migrated. While migrating a dashboard could be done there is a potential that some stuff would be lost.

Extensions are not automatically installed. Besides the security issues, the client API at the time the tool was written didn’t support it. The newer API has support for extensions so a processor could be written to handle this.

Uses are not migrated. As already mentioned in the version control portion of the migration, it would probably have been useful to define an identity mapping from TFS to VSTS users. It is likely that all the VSTS users are already set up in AAD so only the mapping would be needed.

Process customizations are not migrated. Realistically any customizations have to be done early in the migration processs. If you, like us, were migrating from XML to Inherited model then a migration does not make sense. Given the customizations that are available and some of the limitations of the Inherited model this would be tough to automate.

Overall we found the tool to be fast and reliable. It took less than a day to migrate our entire system, which we did over the weekend. On Monday morning everyone was using VSTS and work continued as though nothing had changed. There were a few surprises but overall we haven’t had to go back to our TFS system since we migrated.

Download the code on GitHub