Home » excel » excel – Insufficient memory to continue the execution of the program. Creating Xlxs from CSV

excel – Insufficient memory to continue the execution of the program. Creating Xlxs from CSV

Posted by: admin May 14, 2020 Leave a comment

Questions:

I have a script that cycles through a folder and condenses multiple CSVs to one xlsx file with the names of the CSV as worksheets. However, when the script runs as part of a larger script it failes when it refreshes the query.

$Query.Refresh()

On its own the script runs fine, but when added to the larger one it fails. Can anyone advise why this is the case?

Below is the error I get:

Insufficient memory to continue the execution of the program.
At C:\Temp\Scripts\Shares_Complete.psm1:254 char:13
+             $Query.Refresh()
+             ~~~~~~~~~~~~~~~~
    + CategoryInfo          : OperationStopped: (:) [], OutOfMemoryException
    + FullyQualifiedErrorId : System.OutOfMemoryException

I have tried single csv with the same code and still the same result.

$script:SP = "C:\Temp\Servers\"

$script:TP = "C:\Temp\Servers\Pc.txt"
$script:FSCSV = "C:\Temp\Server_Shares\Server Lists\"
$script:Message1 = "Unknown Hosts"
$script:Message2 = "Unable to connect"
$script:Message3 = "Unknown Errors Occurred"
$script:Txt = ".txt"
$script:OT = ".csv"
$script:FSERROR1 = $FSCSV+$Message1+$OT
$script:FSERROR2 = $FSCSV+$Message2+$OT
$script:FSERROR3 = $FSCSV+$Message2+$OT

$script:ERL3 = $E4 + "Shares_Errors_$Date.txt"
$script:ECL1 = $E4 + "Shares_Exceptions1_$Date.txt"
$script:ERL1 = $E4 + "Shares_Errors1_$Date.txt"
$script:ECL3 = $E4 + "Shares_Exceptions_$Date.txt"

function Excel-Write {
    if ($V -eq "1") {
        return
    }

    [System.GC]::Collect()
    $RD = $FSCSV + "*.csv" 
    $CsvDir = $RD 
    $Ma4 = $FSCSV + "All Server Shares for Domain $CH4"
    $csvs = dir -path $CsvDir # Collects all the .csv's from the driectory 
    $FSh = $csvs | Select-Object -First 1
    $FSh = ($FSh -Split "\")[4]
    $FSh = $FSh -replace ".{5}$"
    $FSh
    $outputxls = "$Ma4.xlsx"
    $script:Excel = New-Object -ComObject Excel.Application
    $Excel.DisplayAlerts = $false
    $workbook = $excel.Workbooks.Add()
    # Loops through each CVS, pulling all the data from each one
    foreach ($iCsv in $csvs) {
        $script:iCsv
        $WN = ($iCsv -Split "\")[-1]
        $WN = $WN -replace ".{4}$"
        if ($WN.Length -gt 30) {
            $WN = $WN.Substring(0, [Math]::Min($WN.Length, 20))
        }
        $Excel = New-Object -ComObject Excel.Application
        $Excel.DisplayAlerts = $false
        $Worksheet = $workbook.Worksheets.Add()
        $Worksheet.Name = $WN

        $TxtConnector = ("TEXT;" + $iCsv)
        $Connector = $worksheet.Querytables.Add($txtconnector,$worksheet.Range("A1"))
        $query = $Worksheet.QueryTables.Item($Connector.Name)

        $query.TextfileOtherDelimiter = $Excel.Application.International(5)

        $Query.TextfileParseType = 1
        $Query.TextFileColumnDataTypes = ,2 * $worksheet.Cells.Column.Count
        $query.AdjustColumnWidth = 1

        $Query.Refresh()
        $Query.Delete()
        $Worksheet.Cells.EntireColumn.AutoFit()
        $Worksheet.Rows.Item(1).Font.Bold = $true
        $Worksheet.Rows.Item(1).HorizontalAlignment = -4108
        $Worksheet.Rows.Item(1).Font.Underline = $true
        $Workbook.Save()
    }
    $Empty = $workbook.Worksheets.Item("Sheet1")
    $Empty.Delete()
    $Workbook.SaveAs($outputxls,51)
    $Workbook.Close()
    $Excel.Quit()
    $ObjForm.Close()
    Delete
}

Should continue script and create the xlsx.

How to&Answers:

Looking at your script, it doesn’t surprise me you eventually run out of memory, because you are continouisly creating Com objects and never release them from memory.

Whenever you have created Com objects and finished with them, use these lines to free up the memory:

$Excel.Quit()
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($workbook) | Out-Null
[System.Runtime.Interopservices.Marshal]::ReleaseComObject($Excel) | Out-Null
[System.GC]::Collect()
[System.GC]::WaitForPendingFinalizers()

Also, take a good look at the code.
You are creating a $Script:Excel = New-Object -ComObject excel.application object before the foreach loop but you don’t use that. Instead, you are creating new Excel and workbook objects inside the loop over and over again where there is absolutely no reason for it since you can re-use the one you created before the loop.

As an aside: The following characters are not allowed in excel worksheet names

\/?*[]:

Length limitation is 31 characters.


EDIT


I had a look at your project and especially the Shares_Complete.psm1 file.
Although I’m not willing of course to rewrite your entire project, I do have some remarks that may help you:

  1. [System.Net.Dns]::GetHostByName() is obsolete. Use GetHostEntry()
  2. when done with a Windows form, use $ObjForm.Dispose() to clear it from memory
  3. you do a lot of [System.GC]::Collect(); [System.GC]::WaitForPendingFinalizers() for no reason
  4. Why not use [System.Windows.MessageBox]::Show() instead of using a Com object $a = new-object -comobject wscript.shell. Again you leave that object lingering in memory..
  5. use Join-Path cmdlet instead of $RD = $FSCSV + "*.csv" or $Cop = $FSCSV + "*.csv" constructs
  6. remove invalid characters from Excel worksheet names (replace '[\\/?*:[\]]', '')
  7. use Verb-Noun names for your functions so it becomes clear what they do. Now you have functions like Location, Delete and File that don’t mean anything
  8. you are calling functions before they are defined like in line 65 where you call function Shares. At that point it does not exist yet because the function itself is written in line 69
  9. add [System.Runtime.Interopservices.Marshal]::ReleaseComObject($worksheet) | Out-Null in function Excel-Write
  10. there is no need to use the variable $Excel in script scope ($Script:Excel = New-Object -ComObject excel.application) where it is used only locally to the function.
  11. you may need to look at Excel specifications and limits
  12. fix your indentation of code so it is clear when a loop or if starts and ends
  13. I would recommend using variable names with more meaning. For an outsider or even yourself after a couple of months two-letter variable names become confusing