Enterprise Etc Adventures and Insights of a .Net Developer

 

As I announced some time ago, I have released my natural language processor to Codeplex.  The documentation on the site is fairly complete, but I wanted to take a different approach on my blog by going at it in more a tutorial fashion starting from downloading the binaries and integrating them into your project.  From there, I will be occasionally posting some tips and tricks that show how flexible the engine is and how to get the most out of it.  Let’s have a look at how to get going with the engine.

Obtaining the Binaries

The first step in building a new project with SpeakToMe is to obtain the binaries containing the types we’ll need.  To get them, go to the Codeplex site’s download page and download the binaries release (SpeakToMe_bin_2012_10_19 as of this writing).  Once the zip file is downloaded, extract the files to a place where you can find them, later.  The zip file should contain three assemblies: Microsoft.Practices.Prism.dll, SpeakToMe.Core.dll and SpeakToMe.Speech.dll.

Create a New Project

Start Visual Studio and create a new WPF project.  The new project dialog should look similar to the below.

Click OK to generate the project.

We need to set reference to the SpeakToMe assemblies.  To do that, create a sub-folder in your new project (I called mine ThirdParty) and copy the three assemblies, mentioned above, into the folder.  Next, in the project references, add references to the three assemblies.

Supporting Users and Conversations

There are a couple of pieces of plumbing we need to build before we get into the fun stuff.  One of these is a data layer that knows how to access information about users and conversations.  Why is this necessary?  SpeakToMe has built in support for multiple users and multiple ways for users to access the engine.  If multiple conversations are going on, the engine needs to be able to figure out how to get a response back to the correct user.  If one user is accessing the system via email and another via chat, there must be a way to insure the correct channel is used to send the response.  So, for the purposes tracking this information, some form of data storage must be implemented.

SpeakToMe does not dictate how you do this.  Instead, it provides an interface describing the needed functionality which you must implement.  There are also types the engine expects to work with which are defined for you.  Let’s see how to pull this all together by building the necessary parts.

Creating a Data Store

The example code that is included in the SpeakToMe source download includes an embedded database implementation.  We’ll use the same schema to create a new database on a local instance of SQL Express.

I have created a new database named SpeakToMe and have created three tables in that database named “Users”, “Conversations” and “ConversationHistory”.  The schema looks like this.

Let’s take a moment to discuss the data that will be stored here and what each column holds.  First of all, each table has an integer key column that is auto-incrementing.  Now, starting with the Users table, the column are pretty self-explanatory where the user name is for display in tooling and the first and last name can be used anywhere.  Next, look at the conversation table.  Each entry in this table describes a single conversation with a single user.  The initiated and active columns describe when the conversations started and whether or not it is still active.  In some cases we are able to determine whether the user has ended the conversation and in other situations we cannot.  If a user ends an IM session with us, we can assume the conversation is over.  If we get an email from a user, we can’t determine whether or not we’ll receive another, so conversations for this type of communication tend to just stay open.

NOTE: I won’t be discussing building functionality for communicating over email and IM in this article, but stay tuned!

The Mode and Address columns are used to determine what channel the communication came from.  Mode is defined as an enumeration in the Core assembly, so we know how we got the message.  However, what if a single user initiates two distinct conversations from two different email accounts?  We include address, here, to make the distinction between the two.  Lastly, UserId denotes the user we’re talking to.

The ConversationHistory table contains each individual communication between the user and the system.  Message and MessageDateTime contain the text of the message and when it was sent.  UserInitiated is true if the message came from the user and is false if it originated with the system.  Tag and TagType are used to store a piece of state along with a particular message so it can be retrieved and referenced later on in the conversation.

Be sure to add the foreign keys between the tables so you’ll be able to access table references on your entities.  I find that creating a Database Diagram is an easy way to do this.

 

Creating a Data Access project.

For this example, I’m going to be using the Entity Framework to access data so I need to create a class library project that will contain my data access code and create a conceptual model.

Add a new class library project to the solution you started.  I’m calling mine “SpeakToMe.Sample.Data”.  Next, delete the class1.cs file that is generated for you.

Now, we need to add the Entity Framework to our data project.  We’ll use NuGet to do this by right-clicking on the references note in the project and selecting “Manage NuGet packages…”.  When the dialog appears, enter “entity framework” into the search box.  The list should populate.  Select the entry in the list entitled “EntityFramework” and click the install button.

After the libraries have been installed into your project, close the NuGet dialog.  Now we need to create our model from the database.  Right-click your project and select Add->New Item.  Select “ADO.Net Entity Data Model” and specify a name for your model.  I named mine “SpeakToMe”.  Click the “Add” button.

In the wizard, select “Generate from database” and click next.  Now, click the Create New Connection button and select your server (local for me) and the database you created.

Click the “Test Connection” button to ensure the information is correct.  If so, click OK.  As a convenience, you might want to leave the Save Connection String checkbox checked.  This will create the connectionstring entry in your config file for use later.

Click Next and wait for the database information to populate in the next screen.  Now, drill down to the tables you created and select all of them.

Click “Finish” and your model will be created.

Implementing the IUserData Interface

We need to add a class, now, that implements the IUserData interface.  In your data project, add a class named UserData and add the attributes shown below.

Note that you’ll need to set references to SpeakToMe.Core and System.ComponentModel.Composition in order to be able to successfully compile this class.  As for the attributes on the class, these allow the use of Microsoft Extensibility Framework (MEF).  SpeakToMe uses MEF as a IoC container and the engine will use it to obtain an instance of this class when it needs to.

Now, we need to implement the methods on the interface.  Below is my implementation.

 

Presence

With SpeakToMe, a particular presence is a mode of communication.  In this post, I have already mentioned email and IM.  You could also implement SMS or any other communication protocol you desire.  In this sample, we’re going to create a simple test presence that will serve as the channel between our WPF application and SpeakToMe.  In a later post, I’ll describe how to build an XMPP (IM) presence.

Let’s create another new class library project to hold our presence class.  I’m calling mine SpeakToMe.Sample.Presence.  Delete the Class1.cs file that is created by default and create a new class called TestPresence.  Now add a references to SpeakToMe.Core and System.ComponentModel.Composition and make your class declaration look like this.

A couple of things to note, here.  First of all, you can tell from the class attributes that SpeakToMe is going to use MEF to load an instance of this class when it’s needed.  Also, this class is implementing another interface that is provided in SpeakToMe.Core.  Any class that will serve as another presence for the system will need to be exported this way and will also need to implement this same interface.  This approach allows you to add as many communication mechanisms as you like without having to touch the source code of the library. 

When implementing the interface, the Initialize method should be used to get everything set up for listening to a channel and sending messages through it.  For example, connecting to an email server or an XMPP server.  The IsConnected property should report whether or not the channel is open for communication.  If a connection is lost, for example, the property should return false.  All processing by the way of sending a message into the SpeakToMe engine should be handled in the ProcessCommand method.  This method has a void return type because we will be notified via an event when the response is ready to be sent back to the user.  Lastly, the OnImportsSatisfied will be called when MEF has supplied all the instances we have requested of it.  More on this in a second.  Now, set references to the SpeakToMe.Core, SpeakToMe.Speech and Microsoft.Practices.Prism assemblies we copied into the third party folder earlier.  Here is my implementation of the TestPresence class.

Off the bat, you’ll notice the EventAggregator property and its associated [Import] attribute.  The attribute is one way we can ask MEF for an instance of a type.  If you look at the OnImportsSatisfied method, you’ll see we’re setting up an event listener there.  We want to do this setup here, because we are using an instance of the EventAggregator type and we want to make sure MEF has given it to us before we start using its reference.  For our sample, we’ll be calling ProcessCommand directly from our client and this method will pass the message on to the NLP engine for processing.  When processing is complete, we’ll be called back on the ReplyToChannelEventHandler where we call the passed in callback to send the reply back to the caller.  For other communication types, we might be sending an email or IM message instead of calling the callback.

The Bootstrapper

I mentioned, earlier, that SpeakToMe uses MEF to load types.  MEF provides an easy way to extend a system without having to put explicit code dependencies in place.  If you’re not familiar with MEF and how it works, I’d suggest you take a bit of time to look at the documentation here.

In a nutshell, however, MEF utilizes a catalog of types.  In this case, we need to point MEF at assemblies we want it to look at.  It will go through the types in the assembly and identify any that have an Export attribute on them.  These are then added to the catalog.  Now, when code needs an instance of the type, it can ask MEF to get one from the catalog.

There are many ways to specify to MEF which assemblies to include.  In our case, we’re going to do this in code.  SpeakToMe includes a BootStrapperBase class that automatically loads the assemblies it knows about.  But, hey, we just added those data access and presence projects and we need a way to tell MEF about them.  This is what the Bootstrapper class is for.  I should also note, however, that it is absolutely required that you implement the bootstrapper whether you have additional assemblies to load or not because without doing so, the default assemblies will never be loaded. 

In the WPF application you created at the beginning of this exercise, create a new class and call it Bootstrapper.  You’ll need to set a reference from that project to SpeakToMe.Speech.  Now, have the new class inherit from BootStrapperBase and override the AddCustomAssembliesToCatalog method

 

It is in this overridden method that we will add our custom assemblies like so.

You’ll need to add references to your projects in order to get a clean compile.

The client

Now that we’ve added all the required plumbing, we can start working on the client we’re going to use to communicate with the engine.  I’m going to start with the default MainWindow.xaml file that was created when we created our WPF project and add a ListBox, TextBox and Button to it like this.

The listbox is at the top a will contain our conversation.  The textbox is at the bottom and will be used to specify our input.  The button sends the input to the engine.  Now let’s name the controls and clean up a bit.  Here is the resulting Xaml.

Let’s do some work in the code behind of this window, now.

First, let’s add some code to the constructor to create and initialize our MEF catalog using the bootstrapper class we created.

Now, we need to add an event handler to the button so we can do something when it’s clicked.

Here is the class after adding the event handler.

 

The first thing we do after the button is clicked is to add our message to the system to the listbox.  Next, we use the ServiceLocator to get an instance of our presence class.  The ServiceLocator is a class we can use to explicitly get an instance of a class from MEF.  If you’ll recall, we exported this class and it should be in our catalog.  Once we have an instance, we call ProcessCommand on that instance and pass in the message, a userId and a callback to be called when a response is ready.  All the callback does is add the system’s response to the listbox so we can observe it.

Initializing the Data in the Database

The last task we have to do before testing is to add some data to the database.  Use whatever tools you need to to add a single user to the users table.  You can specify whatever values you like for the columns.  Just make sure that the id you pass in the ProcessCommand call matches your user’s ID.

We also need an active conversation to be manually created for this test.  Create a new record in the Conversations table for this with the initiated value set to any datetime value, active set to true, mode = 0, address null and userId set to the id of the user you created.

One last thing.  When we were creating our entity model, a connection string was placed in the app.config file for the data project.  You’ll need to copy this connection string and paste it into the app.config file of your WPF app.

Testing

That completes the coding for the test.  Be sure to set the WPF application as the startup project and press F5.  The main window should open.  Type “hello” into the textbox and press the button.  After a second or two, you should see “Hello” followed by your user’s first name appear in the listbox.  This is the response from the system.  Congratulations!  You have just created an application that makes use of the SpeakToMe NLP engine.

Conclusion

Actually, this is more of a beginning than a conclusion.  I will be posting several posts pertaining to the SpeakToMe engine in the coming months.  I’ll cover authoring new token classes, building rules and creating an XMPP presence for the engine.  Along the way, I’ll be pointing out specific best practices and tips.  Hope to have you join me.

Once again, I’d like to let everyone know that I welcome and would greatly appreciate any community participation in this effort.  Contact me through comments on this blog or the SpeakToMe Codeplex site for details on getting started.

After a lot of thought, I have decided to make my Natural Language Engine available as an open source project on Codeplex.  My reservations with doing so in the past had to do with my concern over the quality of the code and also the possibility of potentially monetizing the project at some point.  This processor was written alongside my smart home system and as such was both a bit sloppy as a prototype can be and also pretty tightly integrated with the smart home code.

My change of heart came when I realized that a stand-alone NLP engine is of little use to anyone, but once it becomes an interface to a more capable, feature rich system can be worth quite a bit.  To me, it seems better to put the NLP portion of my hobby project out there for the community to improve and innovate so we can all reap the rewards in the projects we build.  I’ve spent some time cleaning up the code and creating clean extension points.  Please have a look at http://SpeakToMe.Codeplex.com

Roadmap

• First of all, I welcome anyone who would like to contribute to the project. Please contact me at ghostwrtrone at yahoo dot com.
• My goal for the project is to both harden the existing project and also to innovate it by adding new features. Among the areas I’d like to see better implemented are the following.
• · Better rule matching logic. Currently, exact matches are required on token types in the rule method signatures. I’d like a fuzzier matching algorithm so variation in input can still be matched to the proper rule.
• · Expanded library of Tokens. There is no way to predict what domain the application will be used in and thus custom tokens will always be required. Adding more general tokens, however, would shorten the path to a completed application.
• · Better implementation of questions. Currently, the library supports a system asking questions of the user to clarify their statements. This implementation basically places a question object in a collection along with an associated expected token list and a callback that is executed when the question is answered. This list specifies the tokens expected to be in the answer. When the answer is specified, the matching question is located, the callback is executed and the question is removed from the collection. This works ok in many situations, but there are flaws with it as well. First of all, if the answer to a question prompts the system to ask another question, the code structure of the nested callbacks is atrocious. Second, if a user has more than one question in the collection that is expecting the same token list in the answer, there is no way for the system to distinguish between the two questions.

I hope you enjoy the library. If you create a really cool project using it, please shoot me an email and tell me about it. Also, to those who wish to join in the effort, welcome. Let’s keep the project moving forward for all.

In part 1 and part 2 of this short series, we looked at AlchemyAPI and the classes they provide that simplify access to their services and we looked at an overview of the architecture of this application.  Today, I want to supply the source code for the app and discuss a bit about how it works.

Source Code