Table Import and Export
Saving Data
We can save a table object to disk in binary mode with the function saveTable, or save to a text file with the function saveText . Compared to saveTable
, saveText
requires much more space and is more time consuming. However, it is convenient if users need to take a quick look at the data. It is recommended for saving small data sets.
To save a table to a DFS database, please first use function createPartitionedTable to create a DFS table, and then use function append! or tableInsert.
The example below shows how efficiently save saves a table with 20 million records to disk, compared with saveText
.
n=20000000
syms=`IBM`C`MS`MSFT`JPM`ORCL`BIDU`SOHU`GE`EBAY`GOOG`FORD
timestamp=09:30:00+rand(18000,n)
sym=rand(syms,n)
qty=100*(1+rand(100,n))
price=5.0+rand(100.0,n)
t1=table(timestamp,sym,qty,price);
$ timer saveTable("C:/DolphinDB/Data", t1, `trades);
Time elapsed: 208.963 ms
$ saveText(t1, "C:/DolphinDB/Data/trades.csv");
Time elapsed: 3231.914 ms
Loading Data
We can use the function loadTable to load previously saved data, or use the function loadText to import a text file. loadText
is more time consuming than loadTable
(binary mode).
The example below shows how efficiently loadTable
loads a table with 20 million records, compared with loadText
.
$ n=20000000
$ syms=`IBM`C`MS`MSFT`JPM`ORCL`BIDU`SOHU`GE`EBAY`GOOG`FORD
$ timestamp=09:30:00+rand(18000,n)
$ sym=rand(syms,n)
$ qty=100*(1+rand(100,n))
$ price=5.0+rand(100.0,n)
$ t1=table(timestamp,sym,qty,price)
$ saveTable("C:/DolphinDB/Data", t1, `trades);
$ timer tt1 = loadTable("C:/DolphinDB/Data",`trades,,true);
Time elapsed: 179.423 ms
$ saveText(t1, "C:/DolphinDB/Data/trades.txt");
$ timer tt2=loadText("C:/DolphinDB/Data/trades.txt");
Time elapsed: 7609.7 ms