Overview of Sesame: Enhancing Conversational Voice Technology

Sesame is dedicated to advancing the field of conversational voice technology. Their primary focus is on achieving "voice presence," a quality that imbues spoken interactions with a sense of reality, understanding, and value. This is achieved through the development of digital companions that engage in meaningful dialogue, fostering confidence and trust over time.

Key Features

Emotional Intelligence

• Reading and Responding: Sesame's technology is designed to interpret and react to emotional contexts, enhancing the interaction quality between humans and digital assistants.

Conversational Dynamics

• Natural Interaction: The system supports natural timing, pauses, interruptions, and emphasis, mimicking human conversational patterns.

Contextual Awareness

• Adaptive Responses: Adjustments in tone and style are made to suit different situations, ensuring that the voice assistant remains relevant and sensitive to the context of the conversation.

Consistent Personality

• Reliable Interaction: Maintains a coherent and appropriate presence throughout interactions, which helps in building a lasting relationship with users.

Technical Innovations

Conversational Speech Model (CSM)

• End-to-End Learning: Utilizes a single-stage model architecture for improved efficiency and expressivity in voice synthesis.
• Multimodal Approach: Integrates both text and speech inputs using transformers to enhance the contextual understanding of the conversation.

Training and Evaluation

• Amortized Training Process: Reduces the memory load during training by focusing on a subset of audio frames, which preserves the quality of voice generation.
• Evaluation Suite: Developed to assess progress in contextual capabilities, addressing limitations in common public evaluations.

Development and Progress

• Demo Availability: Users can experience the advancements in conversational speech through a demo that showcases friendly and expressive digital companions.
• Research Team: The project is led by a dedicated team including Brendan Iribe, Ankit Kumar, and supported by experts like Johan Schalkwyk, Dan Lyth, and others.

Usage Considerations

• Privacy and Data Handling: Calls are recorded for quality review but are not used for machine learning training and are deleted within 30 days.
• Browser Compatibility: Optimal performance is recommended using Chrome, as audio quality may be degraded in other browsers like iOS/Safari 17.5.

Sesame is actively working towards refining their conversational voice technology to bridge the gap between human and machine interaction, making digital assistants more relatable and effective in everyday tasks.

Related Apps