I just learned that Mysql has a Native CSV storage engine which stores data in a Comma Separated Value file per table.
Is it possible to create a table directly from a uploaded CSV file, something like:
CREATE TABLE USERS < PATH/USERS.CSV
where users.csv is uploaded by the user?
This is not possible. To create table you need a table schema. What you have is a data file. Schema cannot be created with it. What you can do is check if your file has header row. In that case you can create table using that header row but manually.
However there is a way to generate create table statement by a batch file as described by John Swapceinski in the comment section of MySQL manual.
Posted by John Swapceinski on September 5 2011 5:33am.
Create a table using the .csv file’s header:
#!/bin/sh # pass in the file name as an argument: ./mktable filename.csv echo "create table $1 ( " head -1 $1 | sed -e 's/,/ varchar(255),\n/g' echo " varchar(255) );"
Just discovered csvkit, which is a set of unix command line tools for CSV files. Installed on my Mac with
pip install csvkit and the command was:
csvsql --dialect mysql --snifflimit 100000 bigdatafile.csv > maketable.sql
You can alternatively provide a db connection string and it can load the table directly.
I know that this is a little bit stale as a topic, but there is an easier way — IF — you are using phpmyadmin as your mysql front end.
1)Create a database with just default settings.
2)Select the database.
3)Click “Import” at the top of the screen.
4)Select CSV under “Format”.
5)Choose the options appropriate to your csv file (open the csv file in a text editor and reference it to get ‘appropriate’ options).
If you muck it up, no problem, simply drop the database and try again.
In addition to the other solutions mentioned Mac users may want to note that SQL Pro has a CSV import option which works fairly well and is flexible – you can change column names, and field types on import. Choose new table otherwise the initial dialogue can appear somewhat disheartening.
SQL Pro: http://www.sequelpro.com
Sequel Pro – database management application for working with MySQL databases
This is what helped me. Just add your csv file and you are good to go.
if someone is looking for php solutin:
$db = new MySQL_wrapper(MySQL_HOST, MySQL_USER, MySQL_PASS, MySQL_DB); $db->connect(); // this sample gets column names from first row of file //$db->createTableFromCSV('test_files/countrylist.csv', 'csv_to_table_test'); // this sample generates column names $db->createTableFromCSV('test_files/countrylist1.csv', 'csv_to_table_test_no_column_names', ',', '"', '\', 0, array(), 'generate', '\r\n'); /** Create table from CSV file and imports CSV data to Table with possibility to update rows while import. * @param string $file - CSV File path * @param string $table - Table name * @param string $delimiter - COLUMNS TERMINATED BY (Default: ',') * @param string $enclosure - OPTIONALLY ENCLOSED BY (Default: '"') * @param string $escape - ESCAPED BY (Default: '\') * @param integer $ignore - Number of ignored rows (Default: 1) * @param array $update - If row fields needed to be updated eg date format or increment (SQL format only @FIELD is variable with content of that field in CSV row) $update = array('SOME_DATE' => 'STR_TO_DATE(@SOME_DATE, "%d/%m/%Y")', 'SOME_INCREMENT' => '@SOME_INCREMENT + 1') * @param string $getColumnsFrom - Get Columns Names from (file or generate) - this is important if there is update while inserting (Default: file) * @param string $newLine - New line delimiter (Default: \n) * @return number of inserted rows or false */ // function createTableFromCSV($file, $table, $delimiter = ',', $enclosure = '"', $escape = '\', $ignore = 1, $update = array(), $getColumnsFrom = 'file', $newLine = '\r\n') $db->close();
I adopted the script from shiplu.mokadd.im to fit my needs. Whom it interests:
#!/bin/bash if [ "$#" -lt 2 ]; then if [ "$#" -lt 1 ]; then echo "usage: $0 [path to csv file] <table name> > [sql filename]" exit 1 fi TABLENAME=$1 else TABLENAME=$2 fi echo "CREATE TABLE $TABLENAME ( " FIRSTLINE=$(head -1 $1) # convert lowercase characters to uppercase FIRSTLINE=$(echo $FIRSTLINE | tr '[:lower:]' '[:upper:]') # remove spaces FIRSTLINE=$(echo $FIRSTLINE | sed -e 's/ /_/g') # add tab char to the beginning of line FIRSTLINE=$(echo "\t$FIRSTLINE") # add tabs and newline characters FIRSTLINE=$(echo $FIRSTLINE | sed -e 's/,/,\n\t/g') # add VARCHAR FIRSTLINE=$(echo $FIRSTLINE | sed -e 's/,/ VARCHAR(255),/g') # print out result echo -e $FIRSTLINE" VARCHAR(255));"
I’m recommended use MySQL Workbench where is import data. There is also possible make new table from CSV or JSON.
In MySQL Workbench application use context menu on table list and use
Table Data Import Wizard
Link for MySQL Workbench.
I have made a Windows command line tool that do just that.
You can download it here: http://commandline.dk/csv2ddl.htm
C:\Temp>csv2ddl.exe mysql test.csv test.sql
C:\Temp>csv2ddl.exe mysql advanced doublequote comma test.csv test.sql
MySQL for excel plugin can help you.
Open your CSV file in excel. You can use this plugin to export excel data into a new table of remote or local mysql server.
It will analyze your data (top 100 to 1000 rows) and create a corresponding table schema.
This is not possible, you can however overwrite an existing table file. But be sure, that the line endings in your file are unix style (ending only with \n), not windows style (ending with \r\n), whether you are working under windows or not.
If you’re ok with using Python, Pandas worked great for me (csvsql hanged forever for my case). Something like:
from sqlalchemy import create_engine import pandas as pd df = pd.read_csv('/PATH/TO/FILE.csv') # Optional, set your indexes to get Primary Keys df = df.set_index(['COL A', 'COL B']) engine = create_engine('mysql://user:[email protected]/db', echo=False) df.to_sql(table_name, dwh_engine, index=False)
Also this doesn’t solve the “using CSV engine” part which was part of the question but might me useful as well.