Abstract: Distributed stream query is a kind of real-time query computation method based on data stream,which has been widely concerned and developed rapidly in recent years.This paper summarized the research results of the distributed stream processing framework in real-time relational query.There is an in-depth comparison of some products,including the distributed data loading framework,distributed stream computing framework and distributed stream query systems.The paper proposed a distributed stream query model based on Spark Streaming and Apache Kafka,and designed a fast data loading technology based on virtual memory file system,which gets the data loading speed one time faster compare to Apache Flume.On the basis of Spark Streaming,a distributed stream query interface based on Spark SQL was realized,and a method for parsing SQL queries was proposed to implement distributed query in data stream.The experiment results demonstrate that,in the case of complex SQL queries,the method of analyzing SQL by writing code by oneself has obvious advantages.

Key words: Big data,Stream processing system,Distributed stream query,Query optimization,Kafka fast loading