Cluster Mode

Configurations

In cluster mode, DolphinDB configuration files are saved under the config directory.

  • controller.cfg: the configuration file for controller nodes

  • agent.cfg: the configuration file for agent nodes

  • cluster.cfg: the configuration file for data nodes

  • cluster.nodes: network information and node mode of each node in the cluster.

Please note that all configuration files require that the first line is not a blank line.

Configuration Parameters for Controllers

Configuration parameters for controllers can be specified in controller.cfg, or in command line when starting a controller node. localSite and mode must be specified.

Configuration Parameters

Details

localSite

IP address, port number and alias of the controller node.

mode=controller

Node mode. Must specify “controller” here for a controller.

clusterConfig=cluster.cfg

Configuration file for the cluster. The default value is cluster.cfg. It can only be specified in command line.

nodesFile=cluster.nodes

Specify IP address, port number, alias and node mode of all nodes in a cluster. It is loaded when the controller is started. The default value is cluster.nodes.

It can only be specified in command line.

dataSync=0

Whether database logs are forced to persist to disk before the transaction is committed. If dataSync=1, the database log (including redo log, undo log, edit log of data nodes,

and edit log of controller nodes) must be written to disk before each transaction is committed. This guarantees that data written to the database is not lost in case of system

crash or power outage. If dataSync=0, the log files are written to cache before a transaction is committed. The operating system will write the log files to disk at a later time.

We may experience loss of data or corruption of the database in case of system crash or power outage. The default value is 0.

dfsMetaDir=/home/DolphinDB/server

The directory to save the metadata of the distributed file system on the controller node. The default value is the DolphinDB home directory specified by parameter home.

dfsMetaLogFilename=DFSMetaLog

The edit log file for metadata of the distributed file system on the controller node. The default value is DFSMetaLog.

PublicName

Internet IP or domain name of the controller node. It must be a domain name if enableHttps=true.

datanodeRestartInterval

It is a positive integer of int type. Its unit is second. Configure this parameter to implement the following features:

(1) Automatically start the data node after the controller is started (the agent needs to be started)

(2) The data node will be automatically restarted after its offline time is larger than the set value. (agent needs to be started)

The following configuration parameters are for high availability and are specified in controller.cfg.

Configuration Parameters

Details

dfsHAMode=Raft

Whether the controller nodes form a Raft group. It can take the value of either Raft or None. The default value is None.

DfsReplicationFactor=2

The number of all replicas for each data chunk. The default value is 2.

dfsReplicaReliabilityLevel=0

Whether multiple replicas can reside on the same server. 0: Yes; 1: No. 2: replicas are allocated to multiple servers if possible. The default value is 0.

DfsRecoveryWaitTime=0

Length of time (in milliseconds) the controller waits after a data node becomes unavailable before restoring the data to other data nodes. The default value is 0

indicating no automatic recovery.

Configuration Parameters for Agents

Configuration parameters for agents can be specified in agent.cfg, or in command line when starting an agent node. All of the following 3 parameters must be specified.

Configuration Parameters

Details

localSite

Host address, port number and alias of the local node.

mode=agent

Node mode. Must specify “agent” here for an agent.

controllerSite

Host address, port number and alias of the controller site of the agent node. When an agent starts up, it uses this information to contact the controller.

It must be identical as localSite of one  of the controllers in controller.cfg.

lanCluster=1

Whether the cluster is within a LAN (local area network). lanCluster=true: use UDP for heartbeats; lanCluster=false: use TCP for heartbeats. Set lanCluster=false

if the system is deployed in the cloud. The default value is true.

Configuration Parameters for Data Nodes

All configuration parameters in the standalone mode can be used for data nodes in the cluster mode. They are specified in cluster.cfg. Please refer to Standalone Mode for more details.

Streaming-related configuration parameters are also specified in cluster.cfg.

Configuration Parameters

Details

streamingHADir=/home/DolphinDB/Data/NODE1/log/streamLog

The directory to keep streaming Raft log files. The default value is <HOME>/log/streamLog. Each data node should be configured

with different streamingHADir.

streamingHAMode=raft

Enable high-availability for streaming.

streamingRaftGroups=2:NODE1:NODE2:NODE3,3:NODE3:NODE4:NODE5

Information about Raft groups. Each Raft group is represented by group ID and aliases of data nodes in the group, separated with

colon (:). Raft group ID must be an integer greater than 1. Each Raft group has at least 3 data nodes. Use comma (,)

to seperate multiple Raft groups.

LanCluster=1

Whether the cluster is within a LAN (local area network). lanCluster=true: use UDP for heartbeats; lanCluster=false: use TCP

for heartbeats. Set lanCluster=false if the system is deployed in the cloud.

In cluster.cfg, we can specify configuration parameter values in the following 4 ways:

1. Node alias qualified configuration parameters. Node aliases are defined in cluster.nodes.

$ nodeA.volumes = /DFSRoot/nodeA
$ nodeB.volumes = /DFSRoot/nodeB

2. (Node alias + wildcard character (“?” or “%”)) qualified configuration parameters. “?” represents a single character; “%” represents 0, 1 or multiple characters.

$ %8821.volumes = /DFSRoot/data8821
$ %8822.volumes = /DFSRoot/data8822
$ DFSnode?.maxMemSize=16

3. Use macro variable <ALIAS> for assignments with node aliases. For a cluster with 2 data nodes nodeA and nodeB:

$ volumes = /DFSRoot/<ALIAS>

is equivalent to

$ nodeA.volumes = /DFSRoot/nodeA
$ nodeB.volumes = /DFSRoot/nodeB

4. If the parameters are not qualified by node aliases, they indicate common configuration parameter values for all the data nodes in the cluster.

// for Windows
$ maxConnections=64
// for Linux
$ maxConnections=512

$ maxMemSize=12

Parameter Assignment Order

A configuration parameter may appear in the command line or multiple configuration files. It may be assigned different values in different locations. DolphinDB checks for parameter values in the order below. If a parameter is assigned value in a step, then assignments in all subsequent steps are ignored.

1. command line

2. cluster.nodes

3. qualified by specific node aliases in cluster.cfg

4. qualified by node aliases and wildcard characters in cluster.cfg

5. common configuration in cluster.cfg