Memory optimized powershell Out-File -Encoding utf8 without BOM for large files

Question

So I'm running an external command in powershell to pipe mysqldump.exe output into a .sql file.

& "C:\Program Files\MySQL\MySQL Server 5.6\bin\mysqldump.exe" @arguments | Out-File -Encoding utf8 $backupFilePath\$database.sql

Firstly, the file came out in UCS2 encoding. I managed to work out that you can set the encoding on the Out-File command to be -Encoding utf8. But it puts on a Byte Order Mark. Is there any way to explicitly specify that you do not want the Byte Order Mark?

I tried converting the file using WriteAllLines, but this database .sql file output is 3GB in size, and it smashes the memory.

Any thoughts?

I believe this was answered here. http://stackoverflow.com/questions/5596982/using-powershell-to-write-a-file-in-utf-8-without-the-bom — Sunny Chakraborty, Mar 03 '14 at 14:19
Do you have characters outside the ascii range? If not, use `-enc ascii`. — Keith Hill, Mar 03 '14 at 16:40
@SunnyChakraborty As the OP points out, using WriteAllLines isn't a great option for output this large (~3GB). — Keith Hill, Mar 03 '14 at 18:22
Yes, I do have characters outside of the ASCII range. What I ended up doing was calling a batch script from my Powershell Script. Not a very nice solution, but at least it works. If someone has a solution to this, I would very much like to hear it! — Mike, Mar 08 '14 at 00:20
For those who don't know; BOM = Byte-Order Mark. Three chars placed at the beginning of a file (0xEF,0xBB,0xBF) that look like "ï»¿" — Signal15, Nov 26 '14 at 16:51

Mohsen Sarkar · Answer 1 · 2017-02-12T08:02:04.117

Function Out-FileUtf8NoBom {

  [CmdletBinding()]
  param(
    [Parameter(Mandatory, Position=0)] [string] $LiteralPath,
    [switch] $Append,
    [switch] $NoClobber,
    [AllowNull()] [int] $Width,
    [Parameter(ValueFromPipeline)] $InputObject
  )

  [Environment]::CurrentDirectory = $PWD
  $LiteralPath = [IO.Path]::GetFullPath($LiteralPath)

  if ($NoClobber -and (Test-Path $LiteralPath)) { 
    Throw [IO.IOException] "The file '$LiteralPath' already exists."
  }

  $sw = New-Object IO.StreamWriter $LiteralPath, $Append

  $htOutStringArgs = @{}
  if ($Width) {
    $htOutStringArgs += @{ Width = $Width }
  }

  try {
    $Input | Out-String -Stream @htOutStringArgs | % { $sw.WriteLine($_) }
  } finally {
    $sw.Dispose()
  }

}


Function FixEncoding([string] $path)
{
    [IO.SearchOption] $option = [IO.SearchOption]::AllDirectories;
    [String[]] $files = [IO.Directory]::GetFiles((Get-Item $path).FullName, '*.*', $option);
    foreach($file in $files)
    {
        "Converting $file...";
        Get-Content $file | Out-FileUtf8NoBom $file
    }
}

You can use it as FixEncoding("C:\path\to\files\")

Memory optimized powershell Out-File -Encoding utf8 without BOM for large files

1 Answers1