Cortana Integration

Overview

The Cortana integration feature allows you to export your agents in a Cortana compatible VCD format and use it for building apps.

The example code for an integration can be found here.

Below are instructions.

Preparation

Create a Universal Windows Project in Visual Studio.

Add a reference to the API.AI .NET library using Nuget.

Create a new application page and add a microphone/listen button. Then, add TextBlock control to display results.

In the API.AI developer console, go to Integrations from the left side menu and enable Microsoft Cortana integration.

Download a Cortana voice commands file from the API.AI developer console > Agent settings > Export and Import.

Unpack and copy the _W10.xml file to your project. Rename it VoiceCommands.xml. Set the Build Action of the file to Content.

Add a new field of AIService type to the App class.

C#

public AIService AIService { get; private set; }

Initialize the aiService field in the App constructor.

C#

var config = new AIConfiguration("CLIENT_ACCESS_TOKEN", SupportedLanguage.English);
AIService = AIService.CreateService(config);

There are two integration options in Cortana:

  • Register voice command patterns and Cortana will open your app and pass some parameters to it. Official Cortana documentation
  • Register voice command service to provide custom logic for request processing. Voice command service can provide answers without opening your app. Official Cortana documentation

Integration with Voice Commands

Add code to register VoiceCommands.xml at the end of OnLaunched method.

C#

try
{
    var storageFile = await StorageFile.GetFileFromApplicationUriAsync(new Uri("ms-appx:///VoiceCommands.xml"));
    await AIService.InstallVoiceCommands(storageFile);
}
catch (Exception ex)
{
Debug.WriteLine(ex);
}

Implement OnActivated method to receive parameters from Cortana. The API.AI SDK has a pre-built method for processing application launch parameters (passed by Cortana) – AIService.ProcessOnActivatedAsyncfor processing. This method calls API.AI to get an action from API.AI based on speech / text input.

C#

protected async override void OnActivated(IActivatedEventArgs e)
{
    AIResponse aiResponse = null;
    try
    {
        aiResponse = await AIService.ProcessOnActivatedAsync(e);
    }
    catch (Exception)
    {
    // ignored
    }

NavigateToMain(aiResponse);

}

private void NavigateToMain(AIResponse aiResponse) { Frame rootFrame = Window.Current.Content as Frame;

// Do not repeat app initialization when the Window already has content,
// just ensure that the window is active
if (rootFrame == null)
{
    // Create a Frame to act as the navigation context and navigate to the first page
    rootFrame = new Frame();

    rootFrame.NavigationFailed += OnNavigationFailed;

    // Place the frame in the current Window
    Window.Current.Content = rootFrame;
}

rootFrame.Navigate(typeof(MainPage), aiResponse);

// Ensure the current window is active
Window.Current.Activate();

}

In the main application page, add processing for the AIResponse result. For example, output it to a TextBlock.

C#

protected override void OnNavigatedTo(NavigationEventArgs e)
{
    base.OnNavigatedTo(e);

    var response = e.Parameter as AIResponse;
    if (response != null)
    {
        var aiResponse = response;
        resultTextBlock.Text = JsonConvert.SerializeObject(aiResponse, Formatting.Indented);
    }
}

Integration with Voice Command Service

Add Windows Runtime Component project to the solution.

In the Extensions node of your application Package.appxmanifest add the following XML tags.

XML

<Extensions>
    <uap:Extension Category="windows.appService" 
        EntryPoint="ApiAiDemo.VoiceCommands.ApiAiVoiceCommandService">
      <uap:AppService Name="ApiAiVoiceCommandService" />
    </uap:Extension>
    <uap:Extension Category="windows.personalAssistantLaunch"/>
</Extensions>

Add class ApiAiVoiceCommandService to your Windows Runtime Component and add the following code to your VoiceCommandService class.

C#

public sealed class ApiAiVoiceCommandService : IBackgroundTask
{
    public async void Run(IBackgroundTaskInstance taskInstance)
    {
    }
}

Also, you will need BackgroundTaskDeferral to wait until the command is processed, and until VoiceCommandServiceConnection interacts with Cortana and ApiAi interacts with the API.AI service.

C#

private BackgroundTaskDeferral serviceDeferral;
private VoiceCommandServiceConnection voiceServiceConnection;
private ApiAi apiAi;

In the Run method we will process Cortana request using the following steps.

Store BackgroundTaskDeferral instance to wait until work will be completed.

C#

serviceDeferral = taskInstance.GetDeferral();
taskInstance.Canceled += OnTaskCanceled;

Initialize ApiAi instance.

C#

var config = new AIConfiguration("YOUR_CLIENT_ACCESS_TOKEN",
                SupportedLanguage.English);

apiAi = new ApiAi(config);

Get AppServiceTriggerDetails to get VoiceCommandServiceConnection.

C#

var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;

if (triggerDetails != null) { voiceServiceConnection = VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails); voiceServiceConnection.VoiceCommandCompleted += VoiceCommandCompleted; var voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync(); ...

Use VoiceCommandServiceConnection to receive request text and command name.

C#

var recognizedText = voiceCommand.SpeechRecognitionResult?.Text;
var voiceCommandName = voiceCommand.CommandName;

Check the command name for different cases. For example you can:

  • Make requests to API.AI and launch your app with AIResponse from the API.AI.
  • Make request to API.AI and send response to Cortana with SendResponseToCortanaAsync method.
    (e.g. see different processing for "type" and "unknown" voice commands below)

C#

switch (voiceCommandName)
{
    case "type":
        var aiResponse = await apiAi.TextRequestAsync(recognizedText);
        await apiAi.LaunchAppInForegroundAsync(voiceServiceConnection, aiResponse);
        break;
    case "unknown":
        var aiResponse = await apiAi.TextRequestAsync(recognizedText);
        if (aiResponse != null)
        {
        await apiAi.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
        }
        break;
}

Of course, you will need to wrap the entire code with a try...catch block and make task completed in the finally block.

C#

try
{
    ...
}
catch(Exception e)
{
    var message = e.ToString();
    Debug.WriteLine(message);
}
finally
{
    serviceDeferral?.Complete();
}

Full code of the service will look like this.

C#

public sealed class ApiAiVoiceCommandService : IBackgroundTask
{
    private BackgroundTaskDeferral serviceDeferral;
    private VoiceCommandServiceConnection voiceServiceConnection;
    private ApiAi apiAi;

public async void Run(IBackgroundTaskInstance taskInstance)
{
    serviceDeferral = taskInstance.GetDeferral();
    taskInstance.Canceled += OnTaskCanceled;

    var triggerDetails = taskInstance.TriggerDetails as AppServiceTriggerDetails;

    if (triggerDetails != null)
    {
        var config = new AIConfiguration("YOUR_CLIENT_ACCESS_TOKEN", SupportedLanguage.English);

        apiAi = new ApiAi(config);
        apiAi.DataService.PersistSessionId();

        try
        {
            voiceServiceConnection = VoiceCommandServiceConnection.FromAppServiceTriggerDetails(triggerDetails);
            voiceServiceConnection.VoiceCommandCompleted += VoiceCommandCompleted;
            var voiceCommand = await voiceServiceConnection.GetVoiceCommandAsync();
            var recognizedText = voiceCommand.SpeechRecognitionResult?.Text;
            var voiceCommandName = voiceCommand.CommandName;

            switch (voiceCommandName)
            {
                case "type":
                    {
                        var aiResponse = await apiAi.TextRequestAsync(recognizedText);
                        await apiAi.LaunchAppInForegroundAsync(voiceServiceConnection, aiResponse);
                    }
                    break;
                case "unknown":
                    {
                        if (!string.IsNullOrEmpty(recognizedText))
                        {
                            var aiResponse = await apiAi.TextRequestAsync(recognizedText);
                            if (aiResponse != null)
                            {
                                await apiAi.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
                            }
                        }
                    }
                    break;

                case "greetings":
                    {
                        var aiResponse = await apiAi.TextRequestAsync(recognizedText);

                        var repeatMessage = new VoiceCommandUserMessage
                        {
                            DisplayMessage = "Repeat please",
                            SpokenMessage = "Repeat please"
                        };

                        var processingMessage = new VoiceCommandUserMessage
                        {
                            DisplayMessage = aiResponse?.Result?.Fulfillment?.Speech ?? "Pizza",
                            SpokenMessage = ""
                        };

                        var resp = VoiceCommandResponse.CreateResponseForPrompt(processingMessage, repeatMessage);
                        await voiceServiceConnection.ReportSuccessAsync(resp);
                        break;
                    }

                default:
                    if (!<span class="cm-variable-3">string.IsNullOrEmpty(recognizedText))
                    {
                        var aiResponse = await apiAi.TextRequestAsync(recognizedText);
                        if (aiResponse != null)
                        {
                            await apiAi.SendResponseToCortanaAsync(voiceServiceConnection, aiResponse);
                        }
                    }
                    else
                    {
                        await SendResponse("Cannot recognize");
                    }

                    break;
                }

        }
        catch(Exception e)
        {
            var message = e.ToString();
            Debug.WriteLine(message);
        }
        finally
        {
            serviceDeferral?.Complete();
        }
    }
}

private void OnTaskCanceled(IBackgroundTaskInstance sender, BackgroundTaskCancellationReason reason)
{
    serviceDeferral?.Complete();
}

private void VoiceCommandCompleted(VoiceCommandServiceConnection sender, VoiceCommandCompletedEventArgs args)
{
    serviceDeferral?.Complete();
}

private async <span class="cm-variable-3">Task SendResponse(<span class="cm-variable-3">string textResponse)
{
    var userMessage = new VoiceCommandUserMessage
    {
        DisplayMessage = textResponse,
        SpokenMessage = textResponse
    };

    var response = VoiceCommandResponse.CreateResponse(userMessage);
    await voiceServiceConnection.ReportSuccessAsync(response);
}

}