Introduction to RocketMQ
RocketMQ is a pure Java, distributed , queuing model of open source messaging middleware , formerly MetaQ, Ali reference Kafka characteristics of the development of a queuing model of messaging middleware , and then open source to the apache foundation has become apache’s top open source project , with high performance , high reliability , high real-time , distributed features .
Let’s take a look at Ali’s name for him: Rocket. Ali wants him to go to heaven, but I think it’s a pretty cool name.
Let’s take a look at his latest official website
A look back at his journey
2007: Taobao implemented the “Five Colorful Stones” project, which was used to transform the transaction system from a stand-alone to a distributed one, and in the process produced Alibaba’s first-generation messaging engine, Notify. -Notify.
2010: Alibaba B2B department based on ActiveMQ version 5.1 also developed its own messaging engine, called Napoli, this messaging engine is widely used in B2B, not only in the transaction field, in many of the backend asynchronous decoupling and other aspects have also been widely used.
2011: the industry appeared in the Kafka messaging engine is now respected by many big data fields, Alibaba in the study of the overall mechanism and architectural design of Kafka, based on the design of Kafka using Java completely rewritten and launched MetaQ 1.0 version, mainly used to solve the problem of sequential messaging and massive pile up.
2012: Alibaba open source its self-developed third-generation distributed messaging middleware – RocketMQ.
After several years of technical polishing, Ali said that based on RocketMQ technology, the current double eleven day message capacity can reach the trillion level.
November 2016: Ali donates RocketMQ to the Apache Software Foundation, officially becoming an incubation project.
Ali says it will make it a top-tier program. This is a big step for Ali as joining to the Open Source Software Foundation requires assessment and observation by the reviewers.
Frankly speaking, the industry still maintains a stereotypical image of Chinese participation in open source code; of the 342 projects in the Apache Foundation, there are only five projects led by Chinese technologists, namely Kylin, CarbonData, Eagle, Dubbo, and RocketMQ for the time being.
February 20, 2017: RocketMQ officially released version 4.0, experts say that the new version is applicable to the e-commerce field, the financial field, the big data field, and also the programming model of the Internet of Things field.
The above is the overall development history of RocketMQ. In fact, three products have been built around the RocketMQ kernel within Alibaba, namely MetaQ, Notify and Aliware MQ.
Each of these three uses a different model; MetaQ primarily uses a pull model to address sequential messaging and massive stacking; Notify primarily uses a push model to address transactional messaging; and the cloud product, Aliware MQ, offers a commercialized version.
Heroes who have survived many double 11s
In preparation for Double 11 2016, the RocketMq team focused on two things, optimizing for slow requests and unifying the storage engine.
Optimize slow request: Here is the main solution to reduce the jitter and burr of slow request to the whole cluster under the massive high concurrency scenario. This is a very challenging technical work, the team students after more than a month of follow-up tuning, from the double eleven review, 99.996% of the latency fell within 10ms, while 99.6% of the latency within 1ms. Optimization mainly focuses on RocketMQ storage layer algorithm optimization, JVM and operating system tuning. For more details, you can refer to “Distributed Messaging Engine under Trillion Data Flood”.
Unified Storage Engine: It mainly solves the problem of high availability and cost of message engine. Under the premise of multi-generation message engine coexistence, we have fully transplanted and replaced the storage module of Notify.
RocketMQ was born for the financial Internet field, the pursuit of high reliability, high availability, high concurrency, low latency, is an Alibaba from the inside out the success of the breeding model, in addition to thousands of applications in the Ali Group, according to our incomplete statistics, there are at least hundreds of domestic units, scientific research and educational institutions in use.
RocketMQ is also widely used in Ali Group in order, transaction, recharge, streaming computing, message push, log streaming processing, binglog distribution and other scenarios.
Functions he has
It’s probably better if we go straight to GitHub and read Apache’s description of him
Yes the functionality is complete to the point of explosion basically development is completely enough, what? Can’t read the English of specialized vocabulary?
Shuaibing is a warm man, the Chinese function is as follows ↓
- Publish/Subscribe Messaging Model
- Financial Grade Trading News
Various cross-language clients, e.g. Java, C/C++, Python, Go- Pluggable transport protocols, e.g. TCP, SSL, AIO
- Built-in message tracking feature that also supports open tracking
- Versatile big data and streaming ecosystem integration
- Trace messages by time or offset
- Reliable FIFO and strictly ordered messaging in the same queue
- Efficient push and pull consumption modeling
- Cumulative capacity of millions of messages in a single queue
Multiple messaging protocols, such as JMS and OpenMessaging.- Flexible, distributed, horizontally scalable deployment architecture
- Lightning-fast batch message exchange system
- Various message filtering mechanisms, such as SQL and Tag
- Docker Images for Isolation Testing and Cloud Isolation Clusters
- Feature-rich management dashboard for configuration, metrics and monitoring
- Certification and Authorization
What does the composition of his program structure look like?
His core module:
rocketmq-broker: accepts messages from the producer and stores them (by calling rocketmq-store), where the consumer gets the messages from
rocketmq-client: provides the client API for sending and receiving messages.
rocketmq-namesrv: NameServer, similar to Zookeeper, where runtime meta-information such as message TopicName, queues, etc. are kept.
rocketmq-common: some classes, methods, data structures, etc. for common use.
rocketmq-remoting: Netty4-based client/server + fastjson serialization + custom binary protocol.- rocketmq-store: messages, index storage, etc.
rocketmq-filtersrv: message filter Server, it should be noted that to achieve this filtering, you need to upload the code to the MQ! (In general, we utilize the Tag is sufficient to meet most of the filtering needs, if more flexible and more complex filtering needs, you can consider filtersrv component).- rocketmq-tools: command line tools.
The composition of his architecture, or understanding why he is so fast? So strong? So powerful?
He has four main core components: NameServer, Broker, Producer and Consumer.
Tip: We can see that RocketMQ what are clustered deployment, which is one of the reasons for his high throughput, high availability, cluster mode is also very fancy, can support multi-master mode, multi-master multi-slave asynchronous replication mode, multi-master multi-slave synchronous dual-write mode.
And the pattern seems like Kafka ah! (I am nonsense here, itself is Ali based on many features of Kafka development).
Let’s introduce each cluster component separately
NameServer:
Mainly responsible for the management of source data, including the management of Topic and routing information.
NameServer is a full-featured server whose role is similar to Zookeeper in Dubbo, but NameServer is lighter compared to Zookeeper. The main reason is that each NameServer node is independent of each other, without any information interaction.
NameServer will not be too much pressure, usually the main overhead is in maintaining the heartbeat and providing Topic-Broker relationship data.
However, there is a point to note that when the Broker sends a heartbeat to the NameServer, it will bring all the information of the Topics that it is currently responsible for, and if there are too many Topics (10,000 levels), it will lead to dozens of megabytes (M) of data in one heartbeat, and if the network condition is poor, the network transmission fails, and the heartbeat fails, which will lead to NameServer mistakenly thinking that the Broker heartbeat has failed. Broker heartbeat failure.
NameServer is designed to be nearly stateless, horizontally scalable, with nodes that do not communicate with each other and mark themselves as a pseudo-cluster by deploying multiple machines.
Each Broker will register with the NameServer when it starts, the Producer will get the routing information of the Broker from the NameServer according to the Topic before sending the message, and the Consumer will get the routing information of the Topic at regular intervals.
So functionally NameServer should be similar to ZooKeeper. It is said that the early version of RocketMQ did use ZooKeeper, but later changed to its own implementation of NameServer.
Let’s take a look at the role of the registry in Dubbo, is not really the same, the division of the same school similarities really a lot:
Producer
Message producers, responsible for generating messages, are generally responsible for generating messages by the business system.
Producer by the user for distributed deployment, the message from the Producer through a variety of load balancing mode to send to the Broker cluster, send low latency, support for fast failure.
RocketMQ provides three ways to send messages: synchronous, asynchronous, and unidirectional.
Synchronous sending: Synchronous sending means that the sender of a message sends out data and receives a response from the receiver before sending the next packet. Generally used for important notification messages, such as important notification emails, marketing SMS.
Asynchronous sending: Asynchronous sending means that after the sender sends out the data, it does not wait for the receiver to send back the response and then sends out the next packet, which is generally used in business scenarios where the link may take a long time and is sensitive to the response time, such as notifying the start of the transcoding service after a user’s video is uploaded.
One-way sending: one-way sending means only responsible for sending messages without waiting for the server to respond and no callback function trigger, for some very short time-consuming but not high reliability requirements of the scene, such as log collection.
Broker
Message relay role, responsible for storing messages and forwarding them.
Broker is specific to provide business servers , a single Broker node with all the NameServer nodes to maintain a long connection and heartbeat , and will regularly register the Topic information to the NameServer , incidentally the underlying communication and connectivity are based on Netty implementation .
Broker is responsible for message storage to Topic for the latitude to support lightweight queues , stand-alone can support tens of thousands of queue size , support for message push and pull model .
There are data on the official website: with hundreds of millions of message stacking capacity, while strictly guaranteeing the order of messages.
Consumer
Message consumers, responsible for consuming messages, are typically backend systems responsible for asynchronous consumption.
Consumer is also deployed by the user , supports PUSH and PULL two consumption modes , supports cluster consumption and broadcast messages , provides real-time message subscription mechanism .
Pull: Pull Consumer (Pull Consumer) actively pull information from the message server, as long as the bulk pull to the message, the user application will start the consumption process, so Pull is called active consumption type.
Push: Push Consumer encapsulates the pulling of messages, the progress of consumption and other internal maintenance work, leaving the callback interface to be executed when the message arrives to be realized by the user application. So Push is called a passive consumer type, but from the implementation point of view is still pulling messages from the message server, different from Pull is that Push first to register a consumer listener, when the listener is triggered before starting to consume the message.
messaging domain model
Message
Message is the information to be transmitted.
A message must have a Topic, which can be thought of as the address to which your letter is to be mailed.
A message can also have an optional Tag and frontal key-value pairs, which can be used to set a Business Key and look up the message on the Broker to find issues during development.
Topic
Topic (topic) can be regarded as a message class, it is the first level of the message type. For example, an e-commerce system can be divided into: transaction messages, logistics messages, etc., a message must have a Topic.
Topic has a very loose relationship with producers and consumers. A Topic can have 0, 1, or more producers sending messages to it, and a producer can send messages to different Topics at the same time.
A Topic can also be subscribed to by 0, 1, or multiple consumers.
Tag
A Tag can be thought of as a sub-topic, which is a second-level type of message used to provide additional flexibility for the user. Using tags, messages for different purposes in the same business module can be identified by the same Topic but different Tags. For example, transaction messages can be categorized into: transaction creation messages, transaction completion messages, etc. A message can have no Tag.
Tags help keep your code clean and coherent, and can also contribute to the query system provided by RocketMQ.
Group
Grouping, a group can subscribe to multiple Topics.
Divided into ProducerGroup, ConsumerGroup, on behalf of a certain type of producer and consumer, generally speaking, the same service can be used as a Group, the same Group generally speaking, send and consume the same message
Queue
In Kafka called Partition, each Queue is internally ordered, in RocketMQ is divided into read and write two kinds of queues, generally speaking, the number of read and write queues are consistent, if inconsistent will be a lot of problems.
Message Queue
Message Queue, where topics are divided into one or more subtopics, i.e. message queues.
Multiple message queues can be set up under a Topic. When you send a message to a Topic that executes the message, RocketMQ polls all the queues under that Topic to send the message.
The physical management unit of messages. A Topic can have multiple Queue, the introduction of Queue makes the message storage can be distributed clustering, with horizontal scalability.
Offset
In RocketMQ, all message queues are persistent, infinite-length data structures. The so-called infinite-length refers to the fact that each storage unit in the queue is of fixed length, and the storage units in the queue are accessed using the Offset, which is a java long type, 64 bits, and theoretically won’t overflow for 100 years, so it is considered to be infinite in length.
You can also think of a Message Queue as an array of infinite length, with Offset being the subscript.
Message Consumption Patterns
There are two message consumption models: Clustering and Broadcasting.
The default is clustered consumption. In this mode, a cluster of consumers consumes multiple queues of a topic together, and a queue will only be consumed by a single consumer. If a consumer hangs up, the other consumers in the group will continue to consume in place of the one that hanged up.
And broadcast consumption messages are sent to each consumer in the consumer group for consumption.
Message Order
There are two types of Message Order: Orderly (sequential consumption) and Concurrently (parallel consumption).
Sequential consumption means that messages are consumed in the same order as they are sent by the producer for each message queue, so if you are dealing with a scenario where global order is mandatory, you need to make sure that there is only one message queue for the topic being used.
Parallel consumption no longer guarantees message order; the maximum number of parallels consumed is limited by the thread pool specified by each consumer client.
What does a complete communication process look like?
The Producer establishes a long connection with one of the nodes (randomly selected) in the NameServer cluster, periodically obtains Topic routing information from the NameServer, establishes a long connection to the Broker Master that provides Topic service, and sends heartbeats to the Broker at regular intervals.
The Producer can only send messages to the Broker master, but the Consumer is different, it establishes a long connection with both the Master and Slave that provide Topic services, and can subscribe to messages from both the Broker Master and the Broker Slave.
The details are shown below:
I said above that he is like Dubbo is not my blind words, even his registration process is very similar to Dubbo’s service exposure process.
Doesn’t it seem so simple, but at the same time you get curious about how each step is initialized to start?
Shuai C ah know that we are all inquisitive talent, this is not I am ready, let’s analyze it step by step.
It was mainly the boy in the talent group who asked me to write it up. (There is a way to enter the group at the end of the article)
NameService Startup Process
In org.apache.rocketmq.namesrv directory NamesrvStartup this startup class basically describes his startup process we can look at the code:
The first step is to initialize the configuration
Create an instance of NamesrvController and turn on two timed tasks:
Scan the Broker every 10s to remove Brokers that are inactive;Print the KV configuration every 10s.
Step 3 Register the hook function to start the server and listen to the Broker.
NameService there are a lot of things ha I’m here to introduce his startup process, you can also go to see the code, or very interesting, such as route registration will send a heartbeat packet, and heartbeat packet processing process, route deletion, route discovery and so on.
Tip: Originally I wanted to post a lot of source code, after a long discussion with the crooked (Java3y) made the decision not to post, we understand the process-oriented! I mainly do just literacy and some pain points to analyze it, the depth of the study or have to spend time, I want to what are introduced to the space is not enough.
Producer
The link is long and involves a lot of detail, so I’ll just post the link diagram.
Producer is the message sender, so how does he send it?
Load balancing on the sender’s side is achieved by rotating all queues under a certain Topic by the Producer.
Broker
Broker in RocketMQ is to handle the Producer to send message request, Consumer to consume the message request, and message persistence, as well as HA policy and server-side filtering, that is, the cluster in the very heavy work are given to the Broker to deal with.
The Broker module is started via BrokerStartup, which instantiates BrokerController and calls its initialization method
We go to see the Broker’s source code, then you will find that his initialization process is very long, according to the configuration to create a lot of thread pools are mainly used to send messages, pull messages, query messages, client management and consumer management, but also a lot of timed tasks, but also registered a lot of request processors, used to send pull messages to query the message.
Consumer
Let’s not talk about it and just dislike the picture! Dying, next time I’ll just do literacy and write something cool 555
Consumer is message acceptance, so how does he receive messages?
The consumer side will do all the queue loads under the Topic based once in 10 seconds through the RebalanceService thread.
Analysis of Frequently Asked Questions in Interviews
What are his strengths and weaknesses?
RocketMQ：
Stand-alone throughput: 100,000
Availability: very high, distributed architecture
Message reliability: after optimizing the configuration of the parameters, the message can achieve 0 loss
Functional support: MQ function is more complete, or distributed, good scalability
Supports 1 billion level message stacking without performance degradation due to stacking
The source code is java, we can read the source code ourselves and customize our own company’s MQ, we can control the
Born for the financial Internet field, for the high reliability requirements of the scene, especially the e-commerce inside the order deduction, as well as business clipping, in a large number of transactions influx, the back-end may not be able to handle the situation in a timely manner
RoketMQ may be more trustworthy in terms of stability, these business scenarios have been tested many times in Ali Double 11, if your business has the above concurrency scenarios, it is recommended that you can choose RocketMQ
RocketMQ：
There are not many supported client languages, currently java and c++, of which c++ is immature.Community activity is not particularly active
JMS and other interfaces are not implemented in the mq core, and some systems require a lot of code changes to migrate.
Message de-duplication
De-emphasis principle: use business-side logic to maintain idempotence
Idempotency: that is, the user for the same operation initiated by a request or multiple requests for the same result, not because of multiple clicks and the side effects of the database results are unique, immutable.
As long as idempotency is maintained, no matter how many duplicate messages come in, the final result of processing is the same and needs to be implemented on the business side.
De-duplication policy: Ensure that each message is uniquely numbered (e.g., a unique running number), and ensure that message processing successes occur at the same time as the logs in the de-duplication table.
Create a message table, get this message to do the database insert operation. Give this message a unique primary key (primary key) or unique constraints, then even if there is a repeated consumption of the situation, it will lead to a primary key conflict, then no longer process the message.
Message Repeat
The messaging domain has a definition of QoS for message delivery, categorized into:
- At most once
- At least once
- Exactly once
QoS：Quality of Service
Almost all MQ products claim to do this At least once.
Since it is at least once, message duplication can’t be avoided, especially in a distributed network environment.
For example, if the network fails to flash, the ACK fails to return, etc., the acknowledgement is not delivered to the message queue, causing the message queue to not know that it has already consumed the message, and to distribute the message to other consumers again.
Different message queues send different forms of confirmation messages, for example, RabbitMQ is to send an ACK confirmation message, RocketMQ is to return a CONSUME_SUCCESS success flag, Kafka actually has a concept of offset.
RocketMQ does not have a built-in solution for message de-duplication, and support for this in the latest version remains to be confirmed.
Message Availability
When we choose a good clustering mode, then we need to be concerned about how to store and copy the data, RocketMQ provides synchronous and asynchronous strategies for message swiping to satisfy us, when we choose synchronous swiping, if the swipe timeout will return FLUSH_DISK_TIMEOUT, if it is an asynchronous swipe will not return swipe-related information, and if it is an asynchronous swipe will not return swipe-related information. By choosing synchronous flushing, we can maximize the satisfaction of our message will not be lost.
In addition to the storage has a choice after our master-slave synchronization provides both synchronous and asynchronous modes for replication, of course choosing synchronous improves availability, but the RT time for sending messages will drop by about 10%.
RocketMQ uses a hybrid storage structure that shares a log data file (i.e., CommitLog) for all queues under a single instance of Broker.
Kafka, on the other hand, uses a standalone type of storage structure, one file per queue.
Here, Shuai Bing believes that the disadvantage of RocketMQ using a hybrid storage structure is that there will be more random read operations, so the efficiency of reading is low. At the same time, consuming messages needs to rely on ConsumeQueue, which requires some overhead to build the logical consumption queue.
RocketMQ Brushpad Implementation
Broker operates directly on memory (memory-mapped files) for message access, which provides system throughput, but does not prevent data loss when the machine is powered down, so it needs to be persisted to disk.
The final implementations of flushing all use MappedByteBuffer.force() in NIO to write the data in the mapped area to disk, and in the case of synchronous flushing, wait for the write to complete after the Broker writes the message to the CommitLog mapped area.
In terms of asynchrony, it just wakes up the corresponding thread without guaranteeing the timing of execution, and the flow is shown in the figure.
Sequential messages:
Let me briefly describe a simple implementation of RocketMQ that we use.
Tip: Why use RocketMQ as an example of it, this thing is Ali open source, I asked my friends around a lot of companies have to use, so the reader probability is that if I use this as an example of it, the specifics of the details I will be in the back of the respective chapters of the RocketMQ and Kafka to say.
Producer consumers generally need to ensure that the sequential message words, may be a business scenario, such as the creation of the order, payment, shipment, receipt of goods.
So are these things one order number? One order is definitely one order number. That’s easy.
There are multiple queues under a topic, in order to ensure the sending order, RocketMQ provides MessageQueueSelector queue selection mechanism, he has three implementations.
We can use Hash fetching to have the same order sent to the same queue, and then use synchronous sending to send the payment message only if the create message for the same order is sent successfully. In this way, we ensure that the sending is organized.
RocketMQ’s topic within the queuing mechanism can ensure that the storage to meet the FIFO (First Input First Output simply refers to the first-in-first-out), the rest just need to be consumed by the consumer order can be.
RocketMQ only guarantees sequential sending, sequential consumption is guaranteed by the consumer service!!!!
Here is a good understanding, an order you send when you put a queue inside to go, you the same order number Hash is not still the same result, that is certainly a consumer consumption, that order is not guaranteed?
The real order of consumption of different middleware have their own different implementation I will give an example here, we think to understand.
Distributed transactions:
Half Message
It refers to a message that cannot be consumed by the Consumer for the time being. the Producer has successfully sent the message to the Broker, but the message has been marked with the state, and messages in this state are called half-messages. Requires the Producer
The Consumer can only consume the message after the is applied to it.
message retrieval
Due to network flash, producer application restart, etc.. As a result, the Producer has not performed the secondary acknowledgement of Half Message. This is the result of the Brock server scanning at regular intervals.
Proactively ask the Producer side the final status of the message (Commit or Rollback), the message is the message checkback.
Service A first sends a Half Message to the Brock end, carrying the message that Service B is about to ask for +$100.
When Service A knows that the Half Message was sent successfully, then it starts step 3 to execute the local transaction.
Execution of local transactions (there will be three cases 1. execution success. 2. execution failure. 3. network and other reasons for the lack of response)
If the local transaction succeeds, then Product sends a Commit like the Brock server so that Service B can consume the message.
If the local transaction fails, then Product sends a Rollback like the Brock server, then it simply deletes the half-message above.
If there is a delay in returning a failure or success due to network or other reasons, then the RocketMQ callback interface is executed to perform a transaction check.
Message Filtering
- Broker-side message filtering
In the Broker, according to the Consumer’s requirements to do filtering, the advantage is to reduce for the Consumer useless message network transmission. The disadvantage is that it increases the burden of the Broker and is relatively complex to implement. - Consumer-side message filtering
This type of filtering can be fully customized by the application, but the disadvantage is that many useless messages have to be transmitted to the Consumer side.
Broker Buffer
A Broker’s Buffer usually refers to the size of the in-memory Buffer for a queue in the Broker, and such Buffers are usually limited in size.
In addition, RocketMQ does not have the concept of an in-memory Buffer. RocketMQ’s queues are persistent disks, and the data is cleared periodically.
RocketMQ and other MQ has a very significant difference, RocketMQ’s memory buffer is abstracted into an infinite length queue, no matter how much data comes in can be loaded, this infinite is a prerequisite, the Broker will periodically delete the expired data.
For example, if Broker only keeps 3 days of messages, then this Buffer will be deleted from the end of the queue even though the length of this Buffer is infinite, but the data before 3 days will be deleted from the end of the queue.
Retrospective consumption
Retrospective consumption means that Consumer has already consumed the successful message, due to business needs need to re-consume, to support this feature, Broker in the Consumer after the delivery of successful messages, the message still need to be retained. And re-consumption is generally in accordance with the time dimension.
For example, due to the Consumer system failure, after recovery, you need to re-consume the data from 1 hour ago, then the Broker has to provide a mechanism to backtrack the consumption progress according to the time dimension.
RocketMQ supports backward consumption by time, with a time dimension accurate to milliseconds, either forward or backward.
news dump
The main function of the message middleware is asynchronous decoupling, there is an important function is to block the front-end data flood, to ensure the stability of the back-end system, which requires the message middleware has a certain message stacking capabilities, message stacking is divided into the following two cases:
Messages are stacked in the memory buffer, and once the memory buffer is exceeded, the messages can be discarded according to a certain discard policy, as described in the CORBA Notification specification. Suitable for businesses that can tolerate message discarding, in this case, the message stacking capability mainly lies in the memory buffer size, and message stacking, the performance degradation will not be too big, because the amount of data in the memory for the external provision of access to the ability to have a limited impact.
Messages are stacked into persistent storage system, such as DB, KV storage, file record form. When the message can not be hit in the memory Cache, to inevitably access the disk, will produce a large number of read IO, read IO throughput directly determines the message stacked after the access capacity.- There are four main points in assessing message stacking capability:
- How many messages can be stacked and how many bytes? That is, the stacking capacity of the message.
Is the throughput size of outgoing messages, when messages are stacked, affected by the stacking?
Are Consumers who consume normally affected when messages pile up?
What is the throughput when accessing messages stacked on disk after message stacking?
Timed Messages
A timed message is a message that cannot be consumed by the Consumer immediately after it is sent to the Broker, but can only be consumed at a specific point in time or after waiting for a specific amount of time.
If you want to support arbitrary time precision, at the Broker level, you have to do message sorting, and if persistence is involved, then message sorting inevitably incurs a huge performance overhead.
RocketMQ supports timed messages, but does not support arbitrary time precision, it supports specific levels, for example, timed 5s, 10s, 1m, etc.
Write this simple introduction to the tedious middleware, we look at the estimate is also tired, has broken 10,000 words, in the future, I write less of this type, we always let me write a little depth, I say a lot of things I really source code a post, see no one to see.
Kafka I will not post a blog, you can go to GItHub to read the first time, the back will come out how to build the project in the server tutorials, there are some big cattle personal experience and personal book list of things, this year should be the first so write, the main thing is genuinely too busy, hope to understand.