SPJeff

I recently needed to parse a large CSV text file and break it into smaller batches.  Parsing text with PowerShell can easily be done.  The trick here was to manage two pointers $line (within original large text file) and $i (iterate current up to next break threshold).  The first CSV line with column headers from the original parent text file is preserved in all child CSV files. 

Cheers! 

Source Code

# Read parent CSV
$InputFilename = Get-Content '.\source.csv'
$OutputFilenamePattern = 'output_done_'
$LineLimit = 5000
# Initialize
$line = 0
$i = 0
$file = 0
$start = 0
# Loop all text lines
while ($line -le $InputFilename.Length) {
    # Generate child CSVs
    if ($i -eq $LineLimit -Or $line -eq $InputFilename.Length) {
        $file++
        $Filename = "$OutputFilenamePattern$file.csv"
        $InputFilename[$start..($line - 1)] | Out-File $Filename -Force
        $start = $line;
        $i = 0
        Write-Host "$Filename"
    }
    # Increment counters
    $i++;
    $line++
}