Skip to content

examples

Directory actions

More options

Directory actions

More options

Latest commit

 

History

History

examples

Folders and files

NameName
Last commit message
Last commit date

parent directory

..
.gitkeep
.gitkeep
 
 
mutidal_multiturn_vllm_demo.py
mutidal_multiturn_vllm_demo.py
 
 
readme.md
readme.md
 
 
vllm_demo.py
vllm_demo.py
 
 

readme.md

Usage

  • Start the talker vllm service first
MODEL_PATH=YOUR_MODEL_PATH
python talker/talker_vllm_server.py --model ${MODEL_PATH}/talker --gpu-memory-utilization 0.1 --port 8816
  • Run demo
export PYTHONPATH=./:$PYTHONPATH
python3 examples/vllm_demo.py