Abstract: In current scenarios of the big data processing application,the streaming data processing technique is widely used.Message middleware or message queue is usually applied as the data buffer in streaming data processing.Apache Kafka is often used as the data buffer middleware.The performance of Kafka largely determines the overall performance of the application system.In practical applications,the streaming data generated by upstream data sources is usually unstable,and the static data caching strategy cannot adapt to this variable production environment.In view of this problem,if there is a strategy that can dynamically adjust the data cache according to the upstream traffic changes,the adaptability of the system to environment can be enhanced,the real-time processing of streaming data caching can be realized and the throughput performance can also be improved.In the dynamic caching strategy,a method of monitoring the upstream data traffic is proposed,and the ARIMA model is used to predict the future traffic of data streaming,so as to adjust the settings of streaming data storage in advance.The optimum setting parameter of streaming data cache comes from multi-objective optimization of the experimental results of middleware system performance under various pressures.Comparative experimental results show that,during the peak period of streaming data,the strategy can improve the throughput performance of Apache Kafka by more than 150% while guaranteeing a certain maximum delay,thus the overall performance of the message middleware system can be improved.

Key words: Apache Kafka, Message middleware, Multi-objective optimization, Streaming data processing, Time series forecast