What is the reason for Redis slowing down and how to troubleshoot

2023-01-21 19:48:11

This article mainly introduces the reasons for the slowdown of Redis and how to troubleshoot them. In daily operations, I believe that many people have doubts about the reasons for the slowdown of Redis and how to troubleshoot them. The editor consulted various materials , Sort out simple and easy-to-use operation methods, I hope it will be helpful for you to answer your doubts about why Redis slows down and how to troubleshoot! Next, please follow the editor to learn together! <h3 style="text-align: left;">Reason 1: The instance memory reaches the upper limit</h3>Troubleshooting ideasIf your Redis instance is set If the memory upper limit maxmemory is exceeded, it may also cause Redis to slow down.When we use Redis as a pure cache, we usually set a memory limit maxmemory for this instance, and then set a data elimination strategy. And when the memory of the instance reaches maxmemory, you may find that every time new data is written thereafter, the operation delay becomes larger.Reason for the slowdownWhen the Redis memory reaches maxmemory, before writing new data each time, Redis must be kicked out of the instance For a part of the data, keep the memory of the entire instance below maxmemory before new data can be written in.The logic of kicking out old data also takes time, and the specific time-consuming length depends on the elimination strategy you configure:<ul class=" list-paddingleft-2"><li>allkeys -lru: regardless of whether the key has expired, eliminate the least recently accessed key</li><li>volatile-lru: only eliminate the least recently accessed key with an expiration time</li><li>allkeys-random: Regardless of whether the key is set to expire, the key is randomly eliminated</li><li>volatile-random: Only the expiration time is randomly eliminated key</li><li>allkeys-ttl: regardless of whether the key is set to expire, eliminate the key that is about to expire</li><li>noeviction: do not eliminate For any key, after the instance memory reaches maxmeory, write new data and return an error directly</li><li>allkeys-lfu: Regardless of whether the key is set to expire, the key with the lowest access frequency (4.0 + version support)</li><li>volatile-lfu: only eliminate the most frequently accessedLow, and set the expiration time key (supported by version 4.0+)</li></ul>Which strategy to use, we need to configure it according to the specific business scenario. Generally, the allkeys-lru / volatile-lru elimination strategy is most commonly used. Their processing logic is to randomly take a batch of keys from the instance each time (the number is configurable), and then eliminate a key with the least access, and then remove the remaining The next key is temporarily stored in a pool, and a batch of keys is randomly selected, compared with the keys in the previous pool, and a key with the least access is eliminated. Repeat this until the instance memory drops below maxmemory.It should be noted that the logic of Redis's elimination data is the same as that of deleting expired keys, and it is also executed before the command is actually executed, which means that it will also increase the delay in our operation of Redis. Moreover, writing The higher the OPS, the more noticeable the delay will be.<img src="//img.freeonlinedomain.com/uploads/allimg/20230121/1-230121200QR58.jpg" title="" alt="QQ截图20230121200509_1.jpg"/>In addition, if bigkey is still stored in your Redis instance at this time, it will take a long time to eliminate and delete bigkey to release memory.Did you see it? The hazards of bigkey are everywhere, which is why I reminded you to try not to store bigkey.Solution<ul class=" list-paddingleft-2"><li>Avoid storing bigkeys and reduce the time spent on releasing memory</li><li>Change the elimination strategy to random elimination, which is much faster than LRU (adjust according to business conditions)</li><li>Split the instance and distribute the pressure of key elimination to multiple In the example</li><li>If you are using Redis 4.0 or above, enable the layz-free mechanism, and put the operation of eliminating keys and releasing memory into the background thread (configure lazyfree-lazy- eviction = yes)</li></ul><h3 style="text-align: left;">Reason 2: Open large memory pages</h3>Troubleshooting ideas <ul class=" list-paddingleft-2">We all know that when applications apply for memory from the operating system, they apply for memory pages, and the conventional memory page size is 4KB.<li>Starting from 2.6.38, the Linux kernel supports the large memory page mechanism, which allows applications to apply for memory from the operating system in units of 2MB.</li><li>The memory unit that the application applies to the operating system each time becomes larger, but this also means that it takes longer to apply for memory.</li></ul>Reason for the slowdown<ul class=" list-paddingleft-2"><li>When Redis is executing background RDB and AOF rewrite, it adopts the method of fork child process to deal with. However, after the main process forks the child process, the main process can still receive write requests at this time, and the incoming write requests will use the Copy On Write (copy-on-write) method to operate memory data.</li><li>That is to say, once the main process has data that needs to be modified, Redis will not directly modify the data in the existing memory, but first copy the data in the memory, Then modify the data in this new memory, which is the so-called "copy-on-write".</li><li>You can also understand copy-on-write as, who needs to write, who needs to copy first, and then modify.</li><li>The advantage of this is that any write operation by the parent process will not affect the data persistence of the child process (the child process only persists in the entire instance at the moment of fork All the data can be used, and new data changes are not concerned, because the child process only needs a memory snapshot, which is then persisted to the disk).</li><li>However, please note that when the main process is copying memory data, this stage involves the application of new memory. If the operating system opens large memory pages at this time, During this period, even if the client only modifies 10B of data, Redis will apply to the operating system in units of 2MB when applying for memory, and it will take longer to apply for memory, which will increase the delay of each write request and affect the Redis performance. </li><li>Similarly, if the write request operates on a bigkey, then when the main process copies the bigkey memory block, the memory requested at one time will be larger. It will also take longer. It can be seen that bigkey affects performance again here.</li></ul> SolutionTurn off the memory huge page mechanism.First, you need to check whether the Redis machine has enabled huge pages:$ cat /sys/kernel/mm/transparent_hugepage/enabled [always] madvise neverIf the output option is always, it means that the memory huge page mechanism is currently enabled, we need to turn it off:$ echo never > /sys/kernel/ mm/transparent_hugepage/enabledActually, the advantage of the memory huge page mechanism provided by the operating system is that it can reduce the number of applications for memory in a certain program.But for Redis, a database that is extremely sensitive to performance and latency, we hope that Redis will take as little time as possible when applying for memory each time, so I do not recommend you to enable this mechanism on the Redis machine .<h3 style="text-align: left;">Reason 3: Use Swap</h3>Troubleshoot ideasIf you find that Redis suddenly becomes very slow, each operation consumes If the time reaches hundreds of milliseconds or even seconds, then you need to check whether Redis uses Swap. In this case, Redis basically cannot provide high-performance services.What causes the slowdownWhat is Swap? Why does using Swap cause the performance of Redis to degrade?If you have some understanding of the operating system, you will know that in order to alleviate the impact of insufficient memory on the application, the operating system allows a part of the data in the memory to be swapped to the disk to achieve the memory usage of the application Buffering, these memory data are swapped to the area on the disk, which is Swap.The problem is that when the data in the memory is swapped to the disk, when Redis accesses the data again, it needs to read from the disk, and the speed of accessing the disk is hundreds of times slower than accessing the memory ! Especially for a database like Redis, which has extremely high performance requirements and is extremely sensitive to performance, this operation delay is unacceptable.At this point, you need to check the memory usage of the Redis machine to confirm whether Swap is used. You can check whether the Redis process uses Swap in the following ways:# First find the process ID of Redis $ ps -aux | grep redis-server the # View Redis Swap usage $ cat /proc/$pid/smaps | egrep '^(Swap|Size)'The output is as follows<blockquote>Size:             1256 kB Swap:         0 kB Size: 4 kB Swap: 0 kB Size: 132 kB Swap:    0 kB Size:              63488 kB Swap:                  0 kB Size:                132 kB Swap:                  0 kB Size:              65404 kB Swap:                  0 kB Size: 1921024 kB Swap: 0 kB ...</blockquote>This result will list the memory usage of the Redis process.The size of each line indicates the size of a block of memory used by Redis, and the Swap below the Size indicates how much data has been swapped to the disk for the size of the size of the memory. If the two values are equal, it means The data in this memory has been completely swapped to disk.If only a small amount of data is swapped to the disk, for example, each piece of Swap accounts for a small proportion of the corresponding Size, then the impact is not great. If hundreds of megabytes or even gigabytes of memory are swapped to the disk, then you need to be vigilant. In this case, the performance of Redis will definitely drop sharply.Solution<ul class=" list-paddingleft-2"><li>Increase the memory of the machine so that Redis has enough memory to use</li><li>Organize memory space, release enough memory for Redis to use, and then release Redis Swap, let Redis reuse memory</li></ul>Release Redis Swap process It is usually necessary to restart the instance. In order to avoid the impact of restarting the instance on the business, the master-slave switch is generally performed first, and then the Swap of the old master node is released, and the old master node instance is restarted. After the slave database data synchronization is completed, the master-slave switch is performed. Can.It can be seen that when Redis uses Swap, the performance of Redis at this time basically cannot meet the high performance requirements (you can understand that martial arts are abolished), so you also need to prevent this situation in advance.The way to prevent it is that you need to monitor the memory and Swap usage of the Redis machine, alarm when the memory is insufficient or use Swap, and deal with it in time.<h3 style="text-align: left;">Reason 4: Network bandwidth overload</h3>Troubleshooting ideasIf you have avoided the above scenarios that caused performance problems, and Redis has been running stably for a long time, but after a certain point in time, the operation of Redis suddenly starts to slow down, and it continues , what is the cause of this situation?At this point, you need to check whether the network bandwidth of the Redis machine is overloaded, and whether there is an instance that occupies the network bandwidth of the entire machine.Reason for slowing downWhen the network bandwidth is overloaded, the server will experience packet transmission delay and packet loss at the TCP layer and network layer and so on.The high performance of Redis, in addition to operating memory, lies in network IO. If there is a bottleneck in network IO, it will also seriously affect the performance of Redis.Solution<ul class=" list-paddingleft-2"><li>Timely confirm that the Redis instance is full of network bandwidth. If it belongs to normal business access, it needs to be expanded in time Or the instance has been migrated to avoid affecting other instances of this machine due to the excessive traffic of this instance.</li><li>At the operation and maintenance level, you need to increase the monitoring of various indicators of the Redis machine, including network traffic, and alarm in advance when the network traffic reaches a certain threshold, and confirm and expand in time.</li></ul><h3 style="text-align: left;">Reason 5: Other reasons</h3>1) Frequent short connectionsYour business application , you should use long connections to operate Redis to avoid frequent short connections.Frequent short connections will cause Redis to spend a lot of time on connection establishment and release, and TCP's three-way handshake and four-way handshake will also increase access delay.2) Operation and maintenance monitoringAs I mentioned earlier, if you want to predict the slowdown of Redis in advance, it is essential to do a good job Perfect monitoring.Monitoring is actually the collection of various runtime indicators of Redis. The usual method is that the monitoring program regularly collects Redis INFO information, and then performs data display and alarm according to the status data in the INFO information.What I need to remind you here is that you should not take it lightly when writing some monitoring scripts or using open source monitoring components.When writing monitoring scripts to access Redis, try to use long connections to collect status information to avoid frequent short connections. At the same time, you should also pay attention to controlling the frequency of accessing Redis to avoid affecting business requests.When using some open source monitoring components, it is best to understand the implementation principles of these components, so as toAnd correctly configure these components to prevent bugs in the monitoring components, resulting in a large number of Redis operations in a short period of time, affecting the performance of Redis.It happened to us at that time that when the DBA used some open source components, due to configuration and usage problems, the monitoring program frequently established and disconnected from Redis, resulting in slow Redis response.3) Other programs compete for resourcesThe last thing I need to remind you is that your Redis machine is best dedicated and only used to deploy Redis instances , do not deploy other applications, try to provide a relatively "quiet" environment for Redis, to avoid other programs occupying CPU, memory, and disk resources, resulting in insufficient resources allocated to Redis and being affected.

Knowledge sharing community for developers。

Let more developers benefit from it。

Help developers share knowledge through the Internet。

Website navigation

Recommended platform: Hosting; Database; Internet security; CSS Syntax; JavaScript Tutorial

What is the reason for Redis slowing down and how to troubleshoot

Recommended reading

Website navigation

high perspicacity

How to use Redis to implement distributed locks

What is the reason for Redis slowing down and how to troubleshoot

What are the five ways Redis implements distributed locks

How to use Redis master-slave replication