Abstract: During the dialogue,people naturally use multimodal information,e.g．facial expressions and gestures,in addition to using spoken interaction,to support the content expression．The paper proposed a framework on how to efficiently fuse the multimodal information with human-computer dialog model and finally created a multimodal human-computer dialog system．The paper classified the fused methods into three modes,complementary,mixed and indepen-dent,according to their relations between speech channel and other channels．For the dialog framework,the paper proposed a multimodal dialog management model by combining finite state machine,slot filling method and mixed initiative method．The new module can flexibly process the multimodal information during the dialogue．The paper also proposed a Multimodal Markup Language (MML) to control the action of the virtual human for the dialog system．The MML can help to coordinate the complicated actions among different channels for virtual human．Finally,based on above technologies,the paper created a multimodal dialog system and used it for weather information retrieval service.

Key words: Multimodal fusion,Human computer interaction,Dialog management