haStreamTable

Syntax

haStreamTable(raftGroup, table, tableName, cacheLimit, [keyColumn], [retentionMinutes=1440])

Arguments

raftGroup an integer greater than 1 indicating Raft group ID.

table an empty table object created by function table.

tableName a string indicating the name of the high-availability stream table.

cacheLimit an integer representing the maximum number of rows of the high-availability stream table to be kept in memory. If cacheLimit>100,000, it is automatically adjusted to 100,000.

keyColumn is a string indicating the name of the primary key column.

retentionMinutes is an integer indicating for how long (in terms of minutes) the log file keeps data. The default value is 1440, which means the log file only keeps data in the past 24 hours.

Details

Create a high-availability stream table. To use the function, we must enable high availability for streaming by specifying parameters streamingHAMode and streamingRaftGroups in cluster configuration file cluster.cfg.

As the cluster starts up, the data nodes specified by the configuration parameter streamingRaftGroups form Raft groups. In a Raft group, one data node is the Leader and the rest are Followers. There is a copy of the high-availability stream table on each data node in a Raft group.

After creating the high-availability stream table, subscribe to the high-availability stream table on any of the data nodes in a Raft group and set parameter reconnect of function subscribeTable to “true”. The high-availability stream table on the Leader will publish data. If the Leader node goes down, the system will elect a new Leader to continue publishing data. Subscribers will automatically be connected to the high-availability stream table on the new Leader.

A Raft group can have multiple high-availability streaming tables.

Examples

Suppose streamingRaftGroups=11:NODE1:NODE2:NODE3. Execute the following script on any data node of the Raft group to create a high-availability stream table trades:

$ colNames = `timestamp`sym`qty`price
$ colTypes = [TIMESTAMP,SYMBOL,INT,DOUBLE]
$ t=table(1:0,colNames,colTypes)
$ haStreamTable(11,t,`trades,100000);

Execute the followng script on another node that do not belong to the Raft group (NODE4) to subscribe to table trades, and then save the subscribed data to a distributed database.

$ if(existsDatabase("dfs://stock")){
$    dropDatabase("dfs://stock")
$ }
$ db=database("dfs://stock",VALUE,2018.08.01..2019.12.30)
$ t=table(1:0,`timestamp`sym`qty`price,[TIMESTAMP,SYMBOL,INT,DOUBLE])
$ trades_slave=db.createPartitionedTable(t,`trades_slave,`timestamp);
$ subscribeTable(NODE2,`trades,`sub_trades,-1,append!{trades_slave},true,1000,1,-1,true);

Please note that in the script above, the first parameter of function subscribeTable can be any of NODE1, NODE2 and NODE3. Paramater reconnect must be set to “true”.

Execute the following script on NODE4 to cancel the subscription.

$ unsubscribeTable(NODE2,`trades,`sub_trades);