Convai is a versatile developer platform for designing characters with advanced multimodal perception abilities. These characters are designed to integrate…
Convai is a versatile developer platform for designing characters with advanced multimodal perception abilities. These characters are designed to integrate seamlessly into both the virtual and real worlds.
Whether you’re a creator, game designer, or developer, Convai enables you to quickly modify a non-playable character (NPC), from backstory and knowledge to voice and personality. You can do this through the playground user interface or programmatically through the API. Within minutes, creators can witness their characters responding in their own unique style while embodying a spatially-aware NPC capable of performing a wide range of actions.
Convai believes in pushing the state of the art and enabling its users to experience the best in gaming. Some of the fundamental blocks for realistic AI NPCs are facial animations, lip synchronization, and emotion control for the character’s voices, all of which needs to be generated in real time during gameplay.
Figure 1. Examples of features supported by Convai technologies
Convai tapped into NVIDIA Avatar Cloud Engine (ACE), which delivers AI models and microservices that developers can integrate into their pipelines. They used NVIDIA Audio2Face to power the facial animations of their characters, and NVIDIA Riva for speech-to-text and text-to-speech.
The team is currently planning to leverage NVIDIA NeMo and NVIDIA Triton Inference Server, which is part of the NVIDIA AI Enterprise software platform, to develop and deploy its custom LLMs with the lowest possible latency.
“Generative AI-powered characters in virtual worlds unlock various use cases and experiences that were previously impossible,” said Convai Founder and CEO Purnendu Mukherjee. “Convai is leveraging NVIDIA ACE technologies such as Riva automatic speech recognition and Audio2Face to enable lifelike non-playable characters with low latency response times and high fidelity natural animation.”
Convai also developed an extension in NVIDIA Omniverse that enables users to connect their 3D character assets with intelligent conversation agents. Developers can use this extension to take the backstory and voice they have created in Convai and add it to a customizable 3D character in Omniverse. Once in Omniverse, users can have a conversation with the animated character using their computer’s microphone input.
Video 1. Learn how to use the Convai extension for NVIDIA Omniverse
Heighten immersive interactions with AI-driven Convai characters
Traditional NPCs in gaming suffer from many limitations that directly impact the overall player experience. Examples include rigid conversational patterns and a lack of contextual awareness. To address these limitations, Convai has developed a proprietary AI-driven suite of technologies.
Context-aware responsiveness
This innovative approach consists of enabling adaptive and context-aware NPC responsiveness. Characters are equipped with spatial cognition and scene understanding, enabling dynamic comprehension of their environment. Consequently, NPCs perceive situational elements, objects, characters, and their corresponding states in real time.
This heightened awareness, alongside understanding human commands through NVIDIA Riva ASR, empowers NPCs to identify and interact with objects, engage in conversations about their surroundings, and perform actions based on their motivation or as prompted by the player’s conversations.
Other known issues with traditional NPC interactions often come from rigid scripts and predictable behavior. Convai champions more organic interactions by integrating LLM-driven conversation systems with behavior trees that define the default behaviors of the characters. This leads to more dynamic engagements and intelligent responses, while staying true to the original story and gameplay the designer intended.
Characters feel more lifelike if conversations have consequences involving not only responses but also actions. Examples include interacting with the environment or other characters. Convai NPCs can navigate complex verbal instructions and convert them to natural actions in the game engine. Need to grab a snack from the vending machine? No problem—Convai’s complex action sequences enable NPCs to easily handle multistep tasks.
To get started creating characters that respond dynamically to various actions, check out the Convai tutorials.
Emotional awareness
To make the gaming experience even more immersive, Convai characters can perceive emotions and emote naturally. NPCs are emotionally aware, reflecting their feelings through pertinent facial expressions, voices, and gestures. This adds a layer of realism and immersion, making your NPCs feel genuinely alive as they respond with emotional awareness, which adapts throughout the conversation.
To learn more, check out the Convai tutorial on facial expressions. You can also apply to join the Convai early action program for emotional voices.
NPCs are not limited to interacting with the player—they can also interact with fellow NPCs, making the world come alive with real-time generated dialogue content. Creators can decide the topic and the flow of the conversation between the NPCs, making the conversation pertinent to the world they are crafting, while leaving some flexibility in the storyline for conversations personalized to the player’s experience. For more details, check out the Convai tutorial video.
Summary
Convai’s forward-thinking features and approaches transform NPC interactions and elevate gaming with nuanced storytelling, diverse personalities, emotional responsiveness, and contextual accuracy. Currently, more than 15K people have signed up to use Convai. The latest Convai technology also opens the door for multiple emerging use cases, such as building brand representatives and tutors. NVIDIA showcased Convai and the latest NVIDIA ACE technologies in the NVIDIA Kairos demo that debuted at CES 2024.
Learn more about Convai technologies.
Source:: NVIDIA