Processing batch items in parallel

Question

I have an ItemGroup, and want to process all its items in parallel (using a custom task or an .exe).

• I could write my task/exe to accept the entire ItemGroup and process its items in parallel internally. However, I want this parallelism to work in conjunction with MSBuild's /maxCpuCount param, since otherwise I might end up over-parallelizing.
• This thread says there's no way.
• My testing shows that MSBuild's /maxCpuCount only works for building different projects, not items (see code below)

How can I process items from an ItemGroup in parallel?
Is there a way to author a custom task to work in parallel in conjunction with MSBuild's Parallel support?

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">
  <Target Name="Build" >
    <!-- Runs only once - I guess MSBuild detects it's the same project -->
    <!--<MSBuild Projects="$(MSBuildProjectFullPath);$(MSBuildProjectFullPath)" Targets="Wait3000" BuildInParallel="true" />-->

    <!-- Runs in parallel!. Note that b.targets is a copy of the original a.targets -->
    <MSBuild Projects="$(MSBuildProjectFullPath);b.targets" Targets="Wait3000" BuildInParallel="true" />

    <!-- Runs sequentially -->
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>
    <Wait DurationMs="%(Waits.Identity)" />
  </Target>

  <Target Name="Wait3000">
    <Wait DurationMs="3000" />
  </Target>

  <UsingTask TaskName="Wait" TaskFactory="CodeTaskFactory" AssemblyFile="$(MSBuildToolsPath)\Microsoft.Build.Tasks.v4.0.dll" >
    <ParameterGroup>
      <DurationMs ParameterType="System.Int32" Required="true" />
    </ParameterGroup>
    <Task>
      <Code Type="Fragment" Language="cs">
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  Start  DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
        System.Threading.Thread.Sleep(DurationMs);
        Log.LogMessage(string.Format("{0:HH\\:mm\\:ss\\:fff}  End    DurationMs={1}", DateTime.Now, DurationMs), MessageImportance.High);
      </Code>
    </Task>
  </UsingTask>
</Project>   

Answer 1

I know this is old, but if you get a few minutes, revisit your attempt to use the MSBuild task. Using the Properties and/or AdditionalProperties reserved item metadata elements* will resolve the issue you described in your code sample ("Runs only once - I guess MSBuild detects it's the same project").

The MSBuild file below processes items from an ItemGroup in parallel via MSBuild's parallel support (including /maxCpuCount). It does not use BuildTargetsInParallel from the MSBuild Extension Pack, nor any other custom or inline task.

<Project ToolsVersion="4.0" xmlns="http://schemas.microsoft.com/developer/msbuild/2003">

  <Target Name="Build" >
    <ItemGroup>
      <Waits Include="3000;2000"/>
    </ItemGroup>

    <ItemGroup>
      <ProjectItems Include="$(MSBuildProjectFullPath)">
        <Properties>
          WaitMs=%(Waits.Identity)
        </Properties>
      </ProjectItems>
    </ItemGroup>
    <MSBuild Projects="@(ProjectItems)" Targets="WaitSpecifiedMs" BuildInParallel="true" />
  </Target>

  <Target Name="WaitSpecifiedMs">
    <Wait DurationMs="$(WaitMs)" />
  </Target>

</Project>

* Well-hidden under "Properties Metadata" on the MSBuild Task reference page.

Answer 2

As you said yourself, you can't parallelize on target or task level, you can yield though.

My custom tasks parallelize heavily using TPL, i.e. my base task wrapper has a ForEach wrapper.

public bool ForEach<T>(IEnumerable<T> enumerable, Action<T> action, int max = -1)
{
    return enumerable != null && Parallel.ForEach(enumerable, new ParallelOptions { MaxDegreeOfParallelism = max }, (e, s) =>
    {
        if (Canceled)
            s.Stop();
        if (s.ShouldExitCurrentIteration)
            return;
        action(e);
        Interlocked.Increment(ref _total);
    }).IsCompleted;
}

Typically limit is omitted and managed by .NET itself, with few exception like non-thread safe operations like MSDeploy, deploying SSRS reports that has a config DoS limit of 20 from single IP, or a zip task that degrades heavily if it's more than CPU count even by 1. It's probably not worth reading maxCpuCount and use Environment.ProcessorCount or %NUMBER_OF_PROCESSORS%, but you can try parsing the command line or reflecting on the host object, e.g. my base task class has this method to get all properties, targets, etc. for various extra special global flags.

private void Engine(object host)
{
    var type = host.GetType();
    if (type.FullName != "Microsoft.Build.BackEnd.TaskHost")
    {
        Log.Warn("[Host] {0}", type.AssemblyQualifiedName);
        return;
    }

    var flags = BindingFlags.NonPublic | BindingFlags.Instance;
    var taskLoggingContext = type.GetProperty("LoggingContext", flags).GetValue(host, null);
    var targetLoggingContext = taskLoggingContext.GetType().GetProperty("TargetLoggingContext", flags).GetValue(taskLoggingContext, null);

    ProjectTask = taskLoggingContext.GetType().GetProperty("Task", flags).GetValue(taskLoggingContext, null).To<ProjectTaskInstance>();
    ProjectTarget = targetLoggingContext.GetType().GetProperty("Target", flags).GetValue(targetLoggingContext, null).To<ProjectTargetInstance>();

    var entry = type.GetField("requestEntry", flags).GetValue(host);
    var config = entry.GetType().GetProperty("RequestConfiguration").GetValue(entry, null);

    Project = config.GetType().GetProperty("Project").GetValue(config, null).To<ProjectInstance>();
    Properties = Project.Properties.ToDictionary(p => p.Name, p => p.EvaluatedValue);

Typical task would look something like this using ForEach:

public class Transform : Task
{
    [Required]
    public ITaskItem[] Configs { get; set; }

    protected override void Exec()
    {
        //...

        ForEach(Configs, i =>
        {
            //...
        }, Environment.ProcessorCount);

        //...
    }