Smart NPCs: the application of Generative AI in immersive experiences

Joan Rodrigo Carreras

Expert XR Architect, NAKA Client Team Leader

The introduction of GPT-3 in November 2022 marked a complete revolution in the technological landscape of recent years. We had to swiftly adopt this technology for our products and incorporate it into our value proposition. As we explored how generative AI could assist in NAKA, we identified a clear opportunity: applying this new technology to NAKA's NPCs. This approach allows us to provide a more immersive solution through a more natural, accessible, and effective interaction with users.

In this article, we will review the key milestones in the application of generative AI to NPCs, a concept we refer to as NAKA Smart NPCs.

How do Smart NPCs enhance immersive experiences in NAKA?

Let's delve into the primary levers of improvement in virtual experiences through the application of artificial intelligence in NPCs.

They are a guide to become familiar with NAKA

Smart NPCs have the capability to respond to questions related to NAKA, providing users with a more detailed understanding of how a specific experience functions. Upon entering NAKA, users often may not be familiar with how to interact or carry out certain actions within the virtual environment.

By applying generative AI to NPCs, users can inquire about various aspects, such as how to navigate or receive guidance on specific actions. Armed with information about the user's movements within the experience, their expected goals, and the intended progression, the Smart NPC can effectively guide the user to achieve the objectives of the experience.

They provide specific context for the experience

The application of generative AI to the Smart NPCs developed in NAKA makes it possible for them to respond to questions about the specific experience the user is undergoing, offering essential details and clarifications. All of this is done using natural language, both in posing questions and in responding to the user.

With a deep understanding of the ongoing experience, the Smart NPC can provide more accurate and relevant answers. This contextualization ability enhances understanding and facilitates a more informed and seamless interaction.

For example, it would be possible to engage in a role play: we could instruct the NPC to assume a specific role and interact with the user in a training simulation. Let's imagine the experience is about a fire; we could provide context on the cause of the incident and the goal of the experience, enabling the NPC to take on the role of a neighbor offering assistance in this scenario.

They enale interaction with NAKA

While previously actions in NAKA were limited to the use of a controller, the convergence of AI facilitates interaction through natural language, allowing execution by voice commands. Through these voice commands, users can perform the following actions:

· Interact with gadgets. Users can request to perform an action on a gadget by simply describing the task and the specific gadget they want to act upon. For example, playing, pausing, or muting a video by providing the necessary information for identification in the experience or changing the colour of a car in a virtual dealership.

· Interact with the experience. This enables users to interact with gadgets in a more organic way within the context of the experience. For instance, in a specific employee training scenario, the user could naturally command, and the Smart NPC would understand that the activation of various gadgets is needed for that command to work.

Moreover, NAKA's Smart NPCs not only execute specific actions but also understand the context of their surroundings and match it with the user's needs. The Smart NPC can suggest formations or ensure that the user has completed a prerequisite step before executing the next action.

Provide specific organization context

NAKA's Smart NPCs offer specific context about an organization, capable of responding to questions related to the organization by providing pertinent and specific information. 

This personalized approach enhances the user experience by offering more relevant information, contributing to a more effective and satisfying interaction. Consider a new employee training scenario in a company, where users can inquire with the NPC about common questions during an onboarding process.

They have a user state awareness

In relation to all the above, Smart NPCs have the ability to understand where the user is and what they are specifically doing, assisting them with their needs in that particular moment. They can also detect if the user is stuck at a certain point or has been looking at a specific object for an extended period, offering suggestions on what to do.

For instance, imagine a user spending a considerable amount of time examining a car in a showroom. In such a case, the NPC could provide more technical information about the car or showcase it in different colors if the user might find that interesting.

In the context of a training experience, it could even personalize the training content based on the user's profile, considering factors such as knowledge level or position within the company. For example, if the user asks about the next steps in a process, the NPC can provide guidance on the order in which to execute them, using the user's current context within the experience.

If the NPC notices, for instance, that the user has been looking at an object for an extended period, it could suggest taking an action on that object, guiding them for the most satisfying experience possible. It can also assist in resolving a training scenario by engaging in conversation and providing guidance.

Security in NAKA

Finally, it is worth mentioning one of our priorities when incorporating generative AI into NAKA: ensuring the security of our clients. That is why, instead of using the OpenAI service directly, we choose to employ the Azure version, specifically designed to provide security and confidentiality. Each client has the ability to select the generative AI service that best suits their needs, allowing them to choose the option they consider safest for their organization. Additionally, each client uses their own subscription to the service, ensuring complete isolation from other clients and the NAKA platform itself.

As a non-public platform that requires a corporate account for access, all resources in NAKA are under strict control. Information is protected not only by the user's account but also by a permission system that manages access to each experience and resource. This approach ensures detailed control over information and contributes to maintaining the integrity and security of data in NAKA.

In conclusion, this article has discussed some of the possibilities enabled by NPCs in NAKA enhanced with generative AI. Organizations have a wide range of options to incorporate NPCs into their virtual experiences, customizing them according to their characteristics, goals, and specific needs. By carefully considering the unique features of their brands and business objectives, companies can maximize the potential of Smart NPCs to enhance user engagement and create more immersive and effective virtual experiences.

Subscribe to our newsletter