One minute multi-process millions of data are exported to csv and uploaded to oss.

This article brings you some knowledge about php, which mainly introduces how to export millions of data to csv and upload them to oss in one minute. Let's take a look at it with interested friends, hoping to help you.

The product needs to export the walking logs of some users of a team to a csv file, and it should be in a file, and it should be fast and not too time-consuming. Problem logs often mean a large amount of data. To import into a file, although the amount of data is large, it can be solved, which is nothing more than paging query and writing into csv. Fast speed and short time, can also solve nothing more than multi-process export of multiple files, nothing difficult, but the demand is one file! The solution is to find out the total amount of data to be exported and put it into the batch queue to start multiple consumer queries and write them into multiple csv. Record the specific path and file name of csv written by each queue when delivering the queue. Using the characteristics of the batch queue, after the execution is completed, the multiple csvs recorded in the previous step are merged into one CSV in sequence. After the merger is completed, the multiple CSVs before the merger are deleted, and the merged CSVs are kept and uploaded to oss. After testing, more than 500,000 pieces of data are distributed to each queue in the batch queue, and it takes less than one minute to complete a process. How fast this method is depends on how many consumers and processes the server can start. I use an automatic strategy when testing, and the highest is eight at the same time.

Recommended study: laravel Video Tutorial