A comprehensive ROS 2 package suite for human-robot interaction featuring speech synthesis, speech recognition, head movement control with facial emotions, and display management for robot operators.
Overview • Architecture • Installation • Usage • fbot_hri message and services • Commit History • available emotions • Contributing
fbot_hri is a ROS 2 package suite designed for comprehensive human-robot interaction applications. It provides essential capabilities for robots to communicate naturally with humans through speech, express emotions through facial expressions and head movements, and display information to operators through integrated screen management.
The system consists of four main packages:
fbot_hri/
├── 📁 fbot_speech/ # Speech processing and audio interaction
│ ├── 📁 fbot_speech/ # Core speech nodes (TTS, STT, audio player)
│ ├── 📁 speech_scripts/ # Utility scripts for audio processing
│ └── 📁 audios/ # Audio resources (beep, talk sounds)
├── 📁 fbot_head/ # Head movement and facial emotion control
│ ├── 📁 emotions_bridge/ # Emotion-to-hardware bridge
│ ├── 📁 neck_controller/ # Neck movement control
│ └── 📁 head_firmware/ # Hardware communication layer
├── 📁 fbot_screen/ # Display and UI management for operators
│ ├── 📁 display_node
└── 📁 fbot_speech_msgs/ # Custom ROS message definitions for speech
└── 📁 fbot_hri_bringup/ # Launch configurations and parameter files
- ROS2 Humble
- Python 3.10+
- Ubuntu 22.04
- Dependencies listed in
package.xmlandrequirements.txt
-
Clone the repository into your ROS workspace:
cd ~/fbot_ws/src git clone https://github.com/fbotathome/fbot_hri.git
-
Install dependencies:
cd ~/fbot_ws sudo rosdep init # Skip if already initialized rosdep update rosdep install --from-paths src --ignore-src -r -y pip install -r src/fbot_hri/fbot_speech/requirements.txt
-
Build the workspace:
cd ~/fbot_ws colcon build --packages-select fbot_speech fbot_head fbot_screen fbot_speech_msgs fbot_hri_bringup source install/setup.bash
# Launch speech synthesizer with audio player
ros2 launch fbot_hri_bringup synthesizer_speech.launch.py
# Synthesize speech via service
ros2 service call /fbot_speech/ss/say_something \
fbot_speech_msgs/srv/SynthesizeSpeech "{text: 'Hello, I am Boris!', lang: 'en', force_stream_mode: false}"
# Synthesize speech via topic
ros2 topic pub /fbot_speech/ss/say_something \
fbot_speech_msgs/msg/SynthesizeSpeechMessage "{text: 'Hello World', lang: 'en', force_stream_mode: false}"
# Save synthesizer speech via service
ros2 service call /fbot_speech/ss/save_synthesizer \
fbot_speech_msgs/srv/FileSynthesizer "{text: 'Hello, I am Boris!', output_file: '/save/your/file.wav'}"# Launch speech recognizer
ros2 launch fbot_hri_bringup speech_to_text.launch.py stt_config_file:=fbot_stt_recepcionist.yaml
# Start speech recognition
ros2 service call /fbot_speech/sr/speech_recognizer \
fbot_speech_msgs/srv/SpeechToText "{prompt: 'Say something', lang: 'en'}"# Launch speech recognizer
ros2 launch fbot_hri_bringup riva_recognizer.launch.py
# Start speech recognition
ros2 service call /fbot_speech/sr/asr_recognizer \
fbot_speech_msgs/srv/SpeechToText "{string: ['boosted', 'lm', 'words'], boost: 20, sentence: 'True'}"# Launch audio player
ros2 launch fbot_hri_bringup audio_player.launch.py
# Play audio file
ros2 service call /fbot_speech/ap/audio_player \
fbot_speech_msgs/srv/AudioPlayer "{path: '/path/to/audio.wav'}"
# Play beep sound
ros2 service call /fbot_speech/ap/audio_beep std_srvs/srv/Empty# Launch hotword detection
ros2 launch fbot_hri_bringup hotword_detector.launch.py hotword_config_file:=fbot_hotword_restaurant.yaml
# Monitor hotword detection
ros2 topic echo /fbot_speech/bhd/detected# Launch neck controller
ros2 launch fbot_head neck.launch.py
# Launch head control system
ros2 launch fbot_head emotions.launch.py
# Look at specific 3D point
ros2 service call /lookat_start \
fbot_vision_msgs/srv/LookAtDescription3D "{description: 'person', position: {x: 1.0, y: 0.0, z: 1.5}}"
# Stop looking
ros2 service call /lookat_stop std_srvs/srv/Empty
# Send emotion command
ros2 topic pub /fbot_face/emotion std_msgs/msg/String "{data: 'happy'}"
# Control neck position
ros2 topic pub /updateNeck std_msgs/msg/Float64MultiArray "{data: [180.0, 160.0]}"# Launch display system with Foxglove integration
ros2 launch fbot_screen display.launch.py
# Display text message
ros2 topic pub /display_command std_msgs/msg/String "{data: 'sentence:Hello, I am Boris!\nHow can I help you?'}"
# Display image
ros2 topic pub /display_command std_msgs/msg/String "{data: 'image:/path/to/image.jpg'}"
# Display video
ros2 topic pub /display_command std_msgs/msg/String "{data: 'video:/path/to/video.mp4'}"
# Mirror camera topic
ros2 topic pub /display_command std_msgs/msg/String "{data: 'topic:/camera/image_raw'}"| Topic | Type | Description |
|---|---|---|
/fbot_speech/ss/say_something |
SynthesizeSpeechMessage |
Speech synthesis commands |
/fbot_speech/ap/stream_data |
AudioData |
Audio stream data |
/fbot_speech/bhd/detected |
String |
Hotword detection events |
/fbot_speech/bhd/hot_word |
String |
Hotword configuration |
| Topic | Type | Description |
|---|---|---|
/updateNeck |
Float64MultiArray |
Neck position control |
/boris_head/joint_states |
JointState |
Head joint states |
/fbot_face/emotion |
String |
Facial emotion commands |
| Topic | Type | Description |
|---|---|---|
/display_command |
String |
Display control commands |
/ui_display |
Image |
Display output for visualization |
| Service | Type | Description |
|---|---|---|
/fbot_speech/ss/say_something |
SynthesizeSpeech |
Text-to-speech synthesis |
/fbot_speech/ss/saver_synthesizer |
FileSynthesizer |
Text-to-speech saver synthesizer |
/fbot_speech/sr/speech_recognizer |
SpeechToText |
Speech-to-text recognition |
/fbot_speech/sr/asr_recognizer |
RivaToText |
Speech-to-text recognition with RIVA ASR |
/fbot_speech/ap/audio_player |
AudioPlayer |
Play audio files |
/fbot_speech/ap/audio_player_by_data |
AudioPlayerByData |
Play audio from data |
/fbot_speech/ap/audio_beep |
Empty |
Play beep sound |
/fbot_speech/ap/stream_start |
AudioStreamStart |
Start audio streaming |
/fbot_speech/ap/stream_stop |
Empty |
Stop audio streaming |
| Service | Type | Description |
|---|---|---|
/lookat_start |
LookAtDescription3D |
Start looking at 3D target |
/lookat_stop |
Empty |
Stop looking behavior |
The system supports the following facial emotions:
happy- Happy expressionsad- Sad expressionneutral- Neutral/default expressionsurprised- Surprised expressionangry- Angry expressionsuspicious- Suspicious expressionsleepy- Sleepy expression
This repository was created from scratch on August 20, 2025, with files migrated from the fbot_hri_deprecated branch. The previous repository was a fork of another repository, and we wanted it to be independent.
To view commits history before this date, please access the deprecated repository: fbot_hri_deprecated
- Create a feature branch (
git checkout -b feat/amazing-feature) - Commit your changes (
git commit -m 'Add amazing feature') - Push to the branch (
git push origin feat/amazing-feature) - Open a Pull Request