Home » excel » sql – Copy a million record from excel to database

sql – Copy a million record from excel to database

Posted by: admin May 14, 2020 Leave a comment

Questions:

I have 1 million records in my excel sheet and I want to copy those to oracle database in the fastest way possible.

I am aware of 2 ways one using Batch and other using SQLBulkcopy.

Is there a fastest way to do this apart from these 2 methods?

And also I have a requirement to create a log file which contains the details of records which were not inserted while doing the operation.

How to&Answers:

The fastest way will be to make the excel file type to CSV matching all the columns to the headers of the table and then manually importing the CSV to the table.

Answer:

Fastest way to load comma separated files on the remote server is to use direct path load through sql*ldr.. if on the same server you could use external tables as well..

I would recommend you to load using 2 steps..
first create a sql*ldr load script by following the blog .
Then run the generated script in oracle client..
If the table has nologging then adding the append command before execution of the script will make it even faster..

Having said that, a million rows is too small for oracle and should be able to load under a minute if the table is not heavily indexed..

You can load even faster by loading in parallel, but thats a over kill for just a million rows.

Answer:

You can use any Excel parsing library, like apache-poi
It gives you the ability to read through excel spreadsheets.

Just store each row & column, as well as any other data you need.
Then, transfer all of that data to the server/location you wish to see it on.

Example source from one of my programs using this library:

public boolean read() {
    for (final int sheet : sheets()) {
        final Sheet result = workbook.getSheetAt(sheet);
        if (result != null) {
            results.addAll(results(result));
        }
    }
    if (results().size() > 0) {
        System.out.println("Creating backup database incase any failures occur...");
        createBackup();
        return true;
    }
    return false;
}

public boolean open() {
    try {
        results = new HashSet<E>();
        workbook = new HSSFWorkbook(new POIFSFileSystem(new FileInputStream(new File(location()))));
    } catch (IOException e) {
        System.out.println("IOException getting workbook from speadsheet: " + e.getMessage());
    }
    return results != null && workbook != null;
}