Greatest Practices for Market Knowledge Replay – DZone – Uplaza
Software Development

Greatest Practices for Market Knowledge Replay – DZone – Uplaza

TechnoNews

This tutorial introduces the sensible implementation of the DolphinDB’s replay characteristic. Primarily based on DolphinDB’s streaming replay, distributed database structure, and APIs, you may create a robust software for historic perception and mannequin back-testing, which lets you evaluation conditions of curiosity and enhance future efficiency. By following the directions on this tutorial, you’ll study:

  • The workflow of setting up a market replay answer
  • The optimum information storage plan for information sources
  • The strategies to course of and analyze replayed information by way of APIs
  • The efficiency of the replay performance

Be aware: For detailed directions on how you can replay market information, check with Tutorial: Market Knowledge Replay.

1. Assemble a Market Knowledge Replay Resolution

This chapter guides you thru the method of constructing a complete market information replay answer. Our implementation focuses on three forms of market information: quote tick information, commerce tick information, and snapshots of Degree-2 information. The service provides the next capabilities:

  • Multi-client help: Permits submission of replay requests by way of numerous purchasers (together with C++ and Python API)
  • Concurrent person requests: Helps simultaneous replay requests from a number of customers
  • Multi-source synchronization: Allows orderly replay of a number of information sources concurrently
  • Superior ordering choices: Supplies time-based ordering and additional sequential ordering primarily based on timestamps (e.g., primarily based on file numbers of tick information)
  • Completion sign: Indicators the top of a replay
  • Versatile information consumption: Provides subscription-based entry to course of replayed outcomes

1.1 Structure

The answer we construct has the next structure:

  • Market information ingestion: Each real-time and historic tick information could be imported to the DolphinDB distributed database by way of the DolphinDB API or a plugin.
  • Perform encapsulation: Features used throughout the replay course of could be encapsulated as operate views. Key parameters equivalent to inventory record, replay date, replay fee, and information supply can nonetheless be specified by customers.
  • Consumer request: Customers can replay market information by calling the operate view by DolphinDB purchasers (equivalent to DolphinDB GUI, VS Code Extension, and APIs). Customers can even subscribe to and eat real-time replay outcomes on the shopper aspect. Moreover, multi-user concurrent replay is supported.

1.2 Steps

This part introduces the steps for constructing a market information replay answer, as proven within the following determine:

  • Step 1 (Chapter 2): Design and implement an appropriate distributed database construction in DolphinDB. Create the required tables and import historic market information to function the replay information supply.
  • Step 2 (Chapter 3): Develop a DolphinDB operate view that encapsulates the whole replay course of. This abstraction permits customers to provoke replay requests by merely specifying key parameters, while not having to grasp the small print of DolphinDB’s replay performance.
  • Step 3 (Chapters 4 and 5): Make the most of the DolphinDB API to name the beforehand created operate view from exterior purposes. This permits the execution of replay operations outdoors the DolphinDB setting. Moreover, customers can arrange subscriptions to eat real-time replay outcomes on the shopper aspect.

2. Storage Plan for Market Knowledge

The instance on this tutorial replays three forms of market information: quote tick information, commerce tick information, and snapshots of Degree-2 information.

The next chart shows the information (which is saved in TSDB databases) we use within the instance:

The replay mechanism operates on a basic precept: it retrieves the required tick information from the database, types it chronologically, after which writes it to the stream desk. This course of is closely influenced by the effectivity of database studying and sorting operations, which immediately impacts the general replay pace.

To optimize efficiency, a well-designed partitioning scheme is essential. The partitioning scheme of tables outlined above is tailor-made to frequent replay requests, that are sometimes submitted primarily based on particular dates and shares.

As well as, right here, we make the most of the DolphinDB cluster for storing historic information. The cluster consists of three information nodes and maintains two chunk replicas. The distributed nature of the cluster permits for parallel information entry throughout a number of nodes, considerably boosting learn speeds. By sustaining a number of copies of the information, the system achieves greater resilience in opposition to potential node failures.

3. Consumer-Outlined Features for Replay

This chapter delves into the core functionalities of the replay course of and their implementation particulars. These features are then encapsulated into operate views, which could be referred to as by APIs.

The chart under outlines the features within the instance.

3.1 stkReplay: Replay

The stkReplay operate is used to carry out the replay operation.

The operate definition is as follows:

def stkReplay(stkList, mutable startDate, mutable endDate, replayRate, replayUuid, replayName)
{
    maxCnt = 50
    returnBody = dict(STRING, STRING)
    startDate = datetimeParse(startDate, "yyyyMMdd")
    endDate = datetimeParse(endDate, "yyyyMMdd") + 1
    sortColumn = "ApplSeqNum"
    if(stkList.measurement() > maxCnt)
    {
        returnBody["errorCode"] = "0"
        returnBody["errorMsg"] = "Exceeds the limits for a single replay. The maximum is: " + string(maxCnt)
        return returnBody
    }
    if(measurement(replayName) != 0)
    { 
        for(identify in replayName)
        {
            if(not identify in ["snapshot", "order", "transaction"])
            {
                returnBody["errorCode"] = "0"
                returnBody["errorMsg"] = "Input the correct name of the data source. Cannot recognize: " + identify
                return returnBody
            }
        }
    }
    else
    {
        returnBody["errorCode"] = "0"
        returnBody["errorMsg"] = "Missing data source. Input the correct name of the data source."
        return returnBody
    }
    attempt 
    {
        if(measurement(replayName) == 1 && replayName[0] == "snapshot")
        {
            colName = ["timestamp", "biz_type", "biz_data"]
            colType = [TIMESTAMP, SYMBOL, BLOB]
            sortColumn = "NULL"
        }
        else
        {
            colName = ["timestamp", "biz_type", "biz_data", sortColumn]
            colType = [TIMESTAMP, SYMBOL, BLOB, LONG]
        }
        msgTmp = streamTable(10000000:0, colName, colType)
        tabName = "replay_" + replayUuid
        enableTableShareAndPersistence(desk=msgTmp, tableName=tabName, asynWrite=true, compress=true, cacheSize=10000000, retentionMinutes=60, flushMode=0, preCache=1000000)
        
        timeRS = cutPoints(09:30:00.000..15:00:00.000, 23)
        
        inputDict = dict(replayName, every(dsTb{timeRS, startDate, endDate, stkList}, replayName))
        dateDict = dict(replayName, take(`MDDate, replayName.measurement()))
        timeDict = dict(replayName, take(`MDTime, replayName.measurement()))
        
        jobId = "replay_" + replayUuid
        jobDesc = "replay stock data"
        submitJob(jobId, jobDesc, replayJob{inputDict, tabName, dateDict, timeDict, replayRate, sortColumn})
        returnBody["errorCode"] = "1"
        returnBody["errorMsg"] = "Replay successfully"
        return returnBody
    }
    catch(ex)
    {
        returnBody["errorCode"] = "0"
        returnBody["errorMsg"] = "Exception occurred when replaying: " + ex
        return returnBody
    }
}

Be aware: The operate definition is versatile and could be custom-made to go well with particular necessities.

The parameters handed in will first be validated and formatted:

  • The stkList parameter is proscribed to a most of fifty shares (outlined by maxCnt) for a single replay operation.
  • Parameters startDate and endDate are formatted by startDate and endDate utilizing the datetimeParse operate.
  • replayName is the record of knowledge sources to be replayed. It have to be one in every of “snapshot”, “order”, or “transaction”, on this instance.

If any parameters fail validation, the operate experiences errors (as outlined in returnBody). Upon profitable parameter validation, the operate proceeds to initialize the output desk.

The output desk “msgTmp” is outlined as a heterogeneous stream desk the place the BLOB column shops the serialized results of every replayed file. To optimize reminiscence utilization, it’s shared and persevered with the enableTableShareAndPersistence operate.

When the information supply contains each “transaction” and “order”, an additional kind column could be specified for data with the identical timestamp. On this instance, data in “transaction” and “order” with the identical timestamp are sorted by the file quantity “ApplSeqNum” (See part 3.3 replayJob operate for particulars). In such a case, the output desk should comprise the type column. If the information supply accommodates solely “snapshots”, the type column just isn’t required.

Then, submit the replay job utilizing the submitJob operate. You may be knowledgeable in regards to the standing of the replay job, whether or not it has executed efficiently or encountered an exception (as outlined in returnBody).

3.2 dsTb: Assemble Knowledge Sources

The dsTb operate is used to organize the information sources to be replayed.

The operate definition is as follows:

def dsTb(timeRS, startDate, endDate, stkList, replayName)
{
    if(replayName == "snapshot"){
        tab = loadTable("dfs://Test_snapshot", "snapshot")
 }
 else if(replayName == "order") {
  tab = loadTable("dfs://Test_order", "order")
 }
 else if(replayName == "transaction") {
  tab = loadTable("dfs://Test_transaction", "transaction")
 }
 else {
  return NULL
 }
    ds = replayDS(sqlObj=

The dsTb operate serves as a higher-level wrapper across the built-in replayDS operate.

  • It first evaluates the supplied replayName to find out which dataset to make use of.
  • Primarily based on replayName, it hundreds the corresponding desk object from the database.
  • Utilizing replayDS, it segments the information with stkList within the vary of [startDate, endDate) into multiple data sources based on timeRS, and returns a list of data sources.

The timeRS parameter, which corresponds to the timeRepartitionSchema parameter in the replayDS function, is a vector of temporal types to create finer-grained data sources, ensuring efficient querying of the DFS table and optimal management memory usage.

The timeRS in this example is defined as a variable in the stkReplay function, timeRS = cutPoints(09:30:00.000..15:00:00.000, 23), which means dividing data into trading hours (09:30:00.000..15:00:00.000) into 23 equal parts.

To better understand how dsTb works, execute the following script. For the “order” data source, the records with Security ID “000616.SZ” in trading hours of 2021.12.01 are divided into 3 parts.

timeRS = cutPoints(09:30:00.000..15:00:00.000, 3)
startDate = 2021.12.01
endDate = 2021.12.02
stkList = ['000616.SZ']
replayName = ["order"]
ds = dsTb(timeRS, startDate, endDate, stkList, replayName)

This returns an inventory of metacode objects, every containing an SQL assertion representing these information segments.

DataSource= 00:00:00.000000000,date(MDDate) == 2021.12.01,MDDate >= 2021.12.01 and MDDate 
DataSource= 09:30:00.000,date(MDDate) == 2021.12.01,MDDate >= 2021.12.01 and MDDate 
DataSource= 11:20:00.001,date(MDDate) == 2021.12.01,MDDate >= 2021.12.01 and MDDate 
DataSource= 13:10:00.001,date(MDDate) == 2021.12.01,MDDate >= 2021.12.01 and MDDate 
DataSource= 15:00:00.001,date(MDDate) == 2021.12.01,MDDate >= 2021.12.01 and MDDate

3.3 replayJob: Outline Replay Job

The replayJob operate is used to outline the replay job.

The operate definition is as follows:

def replayJob(inputDict, tabName, dateDict, timeDict, replayRate, sortColumn)
{
    if(sortColumn == "NULL")
    {
        replay(inputTables=inputDict, outputTables=objByName(tabName), dateColumn=dateDict, timeColumn=timeDict, replayRate=int(replayRate), absoluteRate=false, parallelLevel=23)
    }
    else
    {
        replay(inputTables=inputDict, outputTables=objByName(tabName), dateColumn=dateDict, timeColumn=timeDict, replayRate=int(replayRate), absoluteRate=false, parallelLevel=23, sortColumns=sortColumn)    
    }
    createEnd(tabName, sortColumn)
}

The replayJob operate encapsulates the built-in replay operate to carry out an N-to-1 heterogeneous replay. As soon as all required information has been replayed, the createEnd operate is named to assemble and write an finish sign into the replay end result, marking the conclusion of the replay course of with a particular file.

The inputDict, dateDict, and timeDict are outlined as variables within the stkReplay operate:

  • inputDict is a dictionary indicating information sources to be replayed. The upper-order operate every is used to outline the information supply for every replayName.
  • dateDict and timeDict are dictionaries indicating the date and time columns of the information sources, used for information sorting.

The replayJob operate features a sortColumn parameter for added information ordering. If the information supply consists solely of “snapshots” (where ordering is not a concern), sortColumn should be set to NULL. In this case, the built-in replay function is called without its sortColumns parameter. For other data sources (“order” and “transaction”), sortColumn can be utilized to specify an additional column for sorting the information with the identical timestamp.

3.4 createEnd: Sign the Finish

The createEnd operate is used to outline the top sign. It may be optionally outlined.

The operate definition is as follows:

def createEnd(tabName, sortColumn)
{
    dbName = "dfs://End"
    tbName = "endline"
    if(not existsDatabase(dbName))
    {
        db = database(listing=dbName, partitionType=VALUE, partitionScheme=2023.04.03..2023.04.04)
        endTb = desk(2200.01.01T23:59:59.000 as DateTime, `END as level, lengthy(0) as ApplSeqNum)
        endLine = db.createPartitionedTable(desk=endTb, tableName=tbName, partitionColumns=`DateTime)
        endLine.append!(endTb)
    }
     
    ds = replayDS(sqlObj=

The createEnd operate indicators the top of the replay course of, writing a file labeled “end” to the output table (specified by tabName). inputEnd, dateEnd, and timeEnd are dictionaries with the string “finish” as the important thing, which corresponds to the second column (biz_type) within the output desk.

To streamline the parsing and consumption of the output desk, a separate database is created particularly for the top sign. Inside this database, a partitioned desk is established. This desk should embody a time column, with different fields being non-obligatory. A simulated file is written to this desk. This file just isn’t strictly outlined and could be custom-made as wanted. For directions on how you can create a database and desk, check with DolphinDB Tutorial: Distributed Database.

The createEnd operate additionally features a sortColumn parameter to find out whether or not to name the built-in replay operate with or with out its sortColumns parameter. The next determine demonstrates an finish sign within the output desk when sortColumn is ready to NULL.


3.5 Encapsulate Features

The features described above are encapsulated into operate views utilizing the addFunctionView operate, as proven within the following script.

addFunctionView(dsTb)
addFunctionView(createEnd)
addFunctionView(replayJob)
addFunctionView(stkReplay)

4. Replay Knowledge With APIs

By creating operate views, we’ve simplified the interface for API purchasers. Now, we solely have to invoke the operate stkReplay in APIs to provoke the replay course of. The replay course of detailed on this chapter is all constructed upon the above operate view.

4.1 With C++ API

This instance runs on a Linux OS, with a DolphinDB C++ API setting arrange. With executable information compiled, you may run the code from the command line interface. The total script is offered within the Appendices: C++ code.

The C++ utility interacts with the DolphinDB server utilizing a DBConnection object (conn). By means of DBConnection, you may execute scripts and features on the DolphinDB server and switch information in each instructions.

The applying invokes the stkReplay operate on the DolphinDB server utilizing the DBConnection::run operate. The args holds all the required arguments for the stkReplay operate. The operate’s result’s captured in a dictionary named ‘end result’. The applying checks the “errorCode” key of the end result utilizing the get technique. If the errorCode just isn’t 1, it signifies an execution error. On this case, the applying returns an error message and terminates this system.

The next code demonstrates how you can name the stkReplay operate.

DictionarySP end result = conn.run("stkReplay", args);
string errorCode = result->get(Util::createString("errorCode"))->getString();
if (errorCode != "1") 
{
    std::cout getString()

When a number of customers provoke replay operations, to forestall conflicts that would come up from duplicate desk names, it’s essential to assign distinctive names to the stream tables being replayed. Executing the next code can generate a singular identifier for every person. This identifier takes the type of a string, equivalent to “Eq8Jk8Dd0Tw5Ej8D”.

string uuid(int len)
{
    char* str = (char*)malloc(len + 1);
    srand(getpid());
    for (int i = 0; i

4.2 With Python API

This instance runs on a Home windows OS, with a DolphinDB Python API setting arrange. The total script is offered within the Appendices: Python Code.

The Python utility interacts with the DolphinDB server utilizing a session object (s). By means of periods, you may execute scripts and features on the DolphinDB server and switch information in each instructions.

As soon as the required variables (stk_list, start_date, end_date, replay_rate, replay_uuid, and replay_name) have been outlined, the add technique is used to add these objects to the DolphinDB server and the run technique is invoked to execute the stkReplay operate.

The next code demonstrates how you can name the stkReplay operate.

stk_list = ['000616.SZ','000681.SZ']
start_date="20211201"
end_date="20211201"
replay_rate = -1
replay_name = ['snapshot']
s.add({'stk_list':stk_list, 'start_date':start_date, 'end_date':end_date, 'replay_rate':replay_rate, 'replay_uuid':uuidStr, 'replay_name':replay_name})
s.run("stkReplay(stk_list, start_date, end_date, replay_rate, replay_uuid, replay_name)")

When a number of customers provoke replay operations, to forestall conflicts that would come up from duplicate desk names, execute the next code can generate a singular identifier for every person. This identifier takes the type of a string, equivalent to “Eq8Jk8Dd0Tw5Ej8D”.

def uuid(size):
    str=""
    for i in vary(size):
        if(i % 3 == 0):
            str += chr(ord('A') + random.randint(0, os.getpid() + 1) % 26)
        elif(i % 3 == 1):
            str += chr(ord('a') + random.randint(0, os.getpid() + 1) % 26)
        else:
            str += chr(ord('0') + random.randint(0, os.getpid() + 1) % 10)
    return str

uuidStr = uuid(16)

5. Devour Replayed Knowledge With APIs

This chapter introduces how you can eat replayed information with C++ API and Python API.

5.1 With C++ API

5.1.1 Assemble a Deserializer

To deserialize the replayed information (which is saved in a heterogeneous desk), a deserializer must be constructed first. Use the next code to assemble a deserializer:

DictionarySP snap_full_schema = conn.run("loadTable("dfs://Test_snapshot", "snapshot").schema()");
DictionarySP order_full_schema = conn.run("loadTable("dfs://Test_order", "order").schema()");
DictionarySP transac_full_schema = conn.run("loadTable("dfs://Test_transaction", "transaction").schema()");
DictionarySP end_full_schema = conn.run("loadTable("dfs://Finish", "endline").schema()");

unordered_map sym2schema;
sym2schema["snapshot"] = snap_full_schema;
sym2schema["order"] = order_full_schema;
sym2schema["transaction"] = transac_full_schema;
sym2schema["end"] = end_full_schema;
StreamDeserializerSP sdsp = new StreamDeserializer(sym2schema);

The preliminary 4 strains get desk schemata of “snapshot,” “order,” “transaction,” and “end signal”. Throughout replay, the suitable schema is chosen as wanted, with the top being required.

To deal with heterogeneous stream tables, a StreamDeserializer object is created utilizing the sym2schema technique. This method maps symbols (keys for supply tables) to their respective desk schemata. Subsequently, each the information sources and the top sign are deserialized primarily based on these schemata.

For detailed directions, see C++ API Reference Information: Establishing a Deserializer.

5.1.2 Subscribe To Replayed Knowledge

Execute the next code to subscribe to the output desk:

int listenport = 10260;
ThreadedClient threadedClient(listenport);
string tableNameUuid = "replay_" + uuidStr;
auto thread = threadedClient.subscribe(hostName, port, myHandler, tableNameUuid, "stkReplay", 0, true, nullptr, true, 500000, 0.001, false, "admin", "123456", sdsp);
std::cout be a part of();

The place variables:

  • listenport is the subscription port of the single-threaded shopper.
  • tableNameUuid is the identify of the stream desk to be consumed.

We subscribe to the stream desk by threadedClient.subscribe. The thread returned factors to the thread that repeatedly invokes myHandler. It should cease calling myHandler when operate unsubscribe is named on the identical matter.

For detailed directions, see the C++ API Reference Information: Subscription.

5.1.3 Outline the Handler for Processing Knowledge

The threadedClient.subscribe technique is used to subscribe to the heterogeneous stream desk. Throughout this course of, the StreamDeserializerSP occasion deserializes incoming information and routes it to a user-defined operate handler. Customers can customise information processing logic by implementing their very own myHandler operate.

On this specific instance, a fundamental output operation is carried out. When the msg is flagged as “end”, the subscription is terminated utilizing threadedClient.unsubscribe. The implementation of myHandler for this instance is as follows:

lengthy sumcount = 0;
lengthy lengthy starttime = Util::getNanoEpochTime();
auto myHandler = [&](vector msgs) 
{
    for (auto& msg : msgs) 
    {
        std::cout getString() get(0)->measurement();
    }
    lengthy lengthy pace = (Util::getNanoEpochTime() - starttime) / sumcount;
    std::cout

5.1.4 Execute the Utility

After compiling the primary file with the aforementioned code throughout the DolphinDB C++ API setting, you may provoke the “replay -subscription — consumption” course of utilizing particular instructions.

  • To replay information of a single inventory from one desk (“order”) for in the future at most pace:
$  ./important 000616.SZ 20211201 20211201 -1 order
  • To replay information of two (or extra) shares from three tables (“snapshot”, “order”, “transaction”) for in the future at most pace:
$  ./important 000616.SZ,000681.SZ 20211201 20211201 -1 snapshot,order,transaction

5.1.5 Output the Outcome

The next determine shows the end result for replaying two shares (“000616.SZ” & “000681.SZ”) from three tables at most pace:

5.2 With Python API

5.2.1 Assemble a Deserializer

To deserialize the replayed information (which is saved in a heterogeneous desk), a deserializer must be constructed first. Use the next code to assemble a deserializer:

sd = ddb.streamDeserializer({
    'snapshot':  ["dfs://Test_snapshot", "snapshot"],
    'order': ["dfs://Test_order", "order"],
    'transaction': ["dfs://Test_transaction", "transaction"],
    'finish': ["dfs://End", "endline"],
}, s)

To deal with heterogeneous stream tables, a streamDeserializer object is created. The parameter sym2table is outlined as a dictionary object. The keys are the supply tables (i.e., “snapshot,” “order,” “transaction,” and “end signal”) and the values are the schema of every desk. Be aware that in replay, the suitable schema is chosen as wanted, with the top sign schema being required.

For detailed directions, see Python API Reference Information: streamDeserializer.

5.2.2 Subscribe To Replayed Knowledge

Execute the next code to subscribe to the output desk:

s.enableStreaming(0)
s.subscribe(host=hostname, port=portname, handler=myHandler, tableName="replay_"+uuidStr, actionName="replay_stock_data", offset=0, resub=False, msgAsTable=False, streamDeserializer=sd, userName="admin", password="123456")
occasion.wait()

The enableStreaming technique is used to allow streaming information subscription. Then name s.subscribe to create the subscription. Name occasion.wait() to dam the present thread to maintain receiving information within the background.

For detailed directions, see the Python API Reference Information: Subscription.

5.2.3 Outline the Handler for Processing Knowledge

The s.subscribe technique is used to subscribe to the heterogeneous stream desk. Throughout this course of, the streamDeserializer occasion deserializes incoming information and routes it to a user-defined operate handler. Customers can customise information processing logic by implementing their very own myHandler operate.

On this specific instance, a fundamental output operation is carried out. The implementation of myHandler for this instance is as follows:

def myHandler(lst):
    if lst[-1] == "snapshot":
        print("SNAPSHOT: ", lst)
    elif lst[-1] == 'order':
        print("ORDER: ", lst)
    elif lst[-1] == 'transaction':
        print("TRANSACTION: ", lst)
    else:
        print("END: ", lst)
        occasion.set()

5.2.4 Output the Outcome

The next determine shows the end result for replaying two shares (“000616.SZ” & “000681.SZ”) from three tables at most pace:

6. Efficiency Testing

We chosen 50 shares starting from 2021.12.01 to 2021.12.09 for our efficiency testing. The take a look at scripts could be discovered within the Appendices: C++ Check Code.

6.1 Check Surroundings

  • Processor household: Intel(R) Xeon(R) Silver 4216 CPU @ 2.10GHz
  • CPU(s): 64
  • Reminiscence: 503 GB
  • Disk: SSD
  • OS: CentOS Linux launch 7.9.2009 (Core)
  • DolphinDB Server: model 2.00.9.3 (launched on 2023.03.29)

6.2 Concurrent Replays of fifty Shares on One Buying and selling Day

We chosen 50 shares traded on 2021.12.01. A number of cases of the C++ API replay program (described in Chapter 4) had been concurrently initiated within the Linux background to submit concurrent replay duties.

  • Elapsed time: The newest end time — the earliest obtain time throughout all replay duties
  • Replay fee: The sum of all customers’ replay information quantity/the overall elapsed time
  • Common single-user elapsed time: (The sum of begin instances — the sum of finish instances)/the variety of concurrent duties
  • Common single-user replay fee: The quantity of knowledge replayed by a single person/the typical single-user elapsed time

6.3 Replays of fifty Shares Throughout A number of Buying and selling Days

We chosen 50 shares starting from 2021.12.01 to 2021.12.09. The occasion of the C++ API replay program (described in Chapter 4) was initiated within the Linux background to submit a activity for replaying information throughout a number of buying and selling days.

  • Elapsed time = The newest end time — the earliest obtain time
  • Replay fee = Knowledge quantity / the overall elapsed time

7. Growth Surroundings

7.1 DolphinDB Server

  • Server Model: 2.00.9.3 (launched on 2023.03.29)
  • Deployment: standalone mode (see standalone deployment)
  • Configuration: cluster.cfg
maxMemSize=128
maxConnections=5000
workerNum=24
webWorkerNum=2
chunkCacheEngineMemSize=16
newValuePartitionPolicy=add
logLevel=INFO
maxLogSize=512
node1.volumes=/ssd/ssd3/pocTSDB/volumes/node1,/ssd/ssd4/pocTSDB/volumes/node1
node2.volumes=/ssd/ssd5/pocTSDB/volumes/node2,/ssd/ssd6/pocTSDB/volumes/node2
node3.volumes=/ssd/ssd7/pocTSDB/volumes/node3,/ssd/ssd8/pocTSDB/volumes/node3
diskIOConcurrencyLevel=0
node1.redoLogDir=/ssd/ssd3/pocTSDB/redoLog/node1
node2.redoLogDir=/ssd/ssd4/pocTSDB/redoLog/node2
node3.redoLogDir=/ssd/ssd5/pocTSDB/redoLog/node3
node1.chunkMetaDir=/ssd/ssd3/pocTSDB/metaDir/chunkMeta/node1
node2.chunkMetaDir=/ssd/ssd4/pocTSDB/metaDir/chunkMeta/node2
node3.chunkMetaDir=/ssd/ssd5/pocTSDB/metaDir/chunkMeta/node3
node1.persistenceDir=/ssd/ssd6/pocTSDB/persistenceDir/node1
node2.persistenceDir=/ssd/ssd7/pocTSDB/persistenceDir/node2
node3.persistenceDir=/ssd/ssd8/pocTSDB/persistenceDir/node3
maxPubConnections=128
subExecutors=24
subThrottle=1
persistenceWorkerNum=1
node1.subPort=8825
node2.subPort=8826
node3.subPort=8827
maxPartitionNumPerQuery=200000
streamingHAMode=raft
streamingRaftGroups=2:node1:node2:node3
node1.streamingHADir=/ssd/ssd6/pocTSDB/streamingHADir/node1
node2.streamingHADir=/ssd/ssd7/pocTSDB/streamingHADir/node2
node3.streamingHADir=/ssd/ssd8/pocTSDB/streamingHADir/node3
TSDBCacheEngineSize=16
TSDBCacheEngineCompression=false
node1.TSDBRedoLogDir=/ssd/ssd3/pocTSDB/TSDBRedoLogDir/node1
node2.TSDBRedoLogDir=/ssd/ssd4/pocTSDB/TSDBRedoLogDir/node2
node3.TSDBRedoLogDir=/ssd/ssd5/pocTSDB/TSDBRedoLogDir/node3
TSDBLevelFileIndexCacheSize=20
TSDBLevelFileIndexCacheInvalidPercent=0.6
lanCluster=0
enableChunkGranularityConfig=true

Be aware: Modify configuration parameter persistenceDir with your individual path.

7.2 DolphinDB Shopper

  • Processor household: Intel(R) Core(TM) i5–11500 @ 2.70GHz 2.71 GHz
  • CPU(s): 12
  • Reminiscence: 16 GB
  • OS: Home windows 11 Professional
  • DolphinDB GUI Model: 1.30.20.1

See GUI to put in DolphinDB GUI.

7.3 DolphinDB C++ API

  • C++ API Model: release200.9

Be aware: It’s endorsed to put in C++API with the model akin to the DolphinDB server. For instance, set up the API of launch 200 for the DolphinDB server V2.00.9.

7.4 DolphinDB Python API

8. Appendices

  • Script for creating databases and tables:
TAGGED:
Share This Article
Leave a comment

Leave a Reply

Your email address will not be published. Required fields are marked *

Exit mobile version