Tech
18/11/2019
Alexa, what is Natural Language Processing?
We as humans have started using natural language processing commonly in our everyday lives. It’s a digital trend which has emerged alongside the prevalence of AI.
5 mins
We as humans have started using natural language processing commonly in our everyday lives. It’s a digital trend which has emerged alongside the prevalence of AI. It is predicted to be a large part of businesses innovation going forward and we, as consumers will be interacting with and using it on a daily basis in addition to implementing it our products.
Natural language processing is essentially voice or text recognition software also known as NLP. It’s the process of taking a human user output processing it with artificial intelligence, transforming it into something that a computer can process and determine an appropriate action to take. For example, when you talk to you mobile phone and ask siri or the google assistant to write an email for you, this is considered NLP.
For NLP to understand human language the program needs to be trained for it. This usually means processing thousands of items of text or speech in order to get a base understanding of the human language. This includes words, grammar and names in addition to the natural flow of the language.
NLP services are usually trained with text books for example since these have correct spelling and grammar throughout. It’s important to remember that the AI will be trained on a specific language such as England or Spanish so if it’s to be used in another language / culture it we need to be trained specifically for it. It can be trained on anything, you can make up your own language and then train the NLP model to respond in the way you want but it might not be a very popular service.
For example “I would like to buy size twelve shoes”.
The intent might be “purchase item” and the entities would be “size 12” and “shoes”.
If this sentence was used with a bot on ASOS for example it may then direct the user to a page of shoes size 12 that they can purchase.
Often NLP models will be designed to work for a specific purpose so they will be expecting a type of intent, this makes it much easier to train and use. For example an ASOS bot will be expecting questions about its products, company or orders and not questions about if the earth is flat.
The NLP software is likely to be running on a server and accepting requests from lots of sources, this means any users with a device connected to the internet should be able to send a request to the software.
Natural language programs that can process human speech usually work by being trained on transforming the voice speech into text. Once they can transform the speech into text they work the same was as other NLP services by processing the text as intent / entities.
A simple use is to direct a user around a website, when they first navigate to a site they might not know where to find what their after so if there is a chatbot on the website they should be able to ask “Where can I find the benefits application form print out” and the bot can reply with a link to the correct page or download.
A very practical use is being able to talk to a GPS in your car, you can ask for directions to the location of the distance left on your journey all via voice speech. This means users don’t have to take their hands off the wheel or their eyes off the road making it much safer.
Another use could be analysing all the posts in a forum, this could generate lots of useful information such as the general sentiment of what users are discussing (games / sports / clothing) their attitude (happy / sad / angry), this feedback could be used to make appropriate improvements. This could also be used to discover anyone using the forums for nefarious uses such as scamming or planning terrorism attacks (yes really!).
There are a couple big players in terms of NLP, most of them are paid services, lets cover them first. This is not a comprehensive list of all the NLP systems available just a few examples.
With the emergence of NLP there has been an explosion in the amount of companies trying to offer their software solution. Several of the big players like Microsoft and Amazon have very capable systems. I would recommend if your looking to experiment and get started in NLP to use Rasa since its open source and has a lot of tutorials and guides on getting started in the world of NLP.
They are usually trained for specific uses so will not be able to handle anything they are not already expecting. For example, if you try asking the ASOS chatbot bot for directions to Leicester, you’re probably not going to get a satisfactory answer.
They also need to be trained for specific languages so if they need to be accessed in an alternate language they will have to be entirely retrained for it.
One issue I encountered when developing a chatbot for a website designed for British Muslims looking to regain their faith was the issue of dual language. I found that often British Muslims will use a mix of English and Islamic words and this caused confusion in the processing. The solution to this was to use a more generic AI to process the input which didn’t rely on knowledge of a specific language but instead used traditional AI techniques of comparing raw characters and words with a very large training data set in order to best match up the user input to a defined intent.
NLP can take a large amount of processing power, training the model to process the inputs can take some time depending on the complexity and the amount of training data. Requesting the model to do the processing on the input can also take a lot of processing power but nowhere near as much as the initial model generation. While this in preventive it is something to consider when developing an NLP system.
When it does go it wrong it can fail spectacularly, for example “how I can I deal with bullies” could be interpreted as the user wants to sell their bike to an bullie rather than how the user should act when confronted by a bully.
A key aspect of the NLP models and technology is that its constantly being improved. As people use and play with the NLP services available, they are being tracked and when a model predicts something correctly or incorrectly, this is being logged down and can be included as training data in the next iteration. As time goes on the NLP services as well as the models we are training are going to get better and better at predicting our language.
As the software and methods for NLP improve over time we will be able to use it in more sensitive areas for example someone’s entire medical history could be processed by NLP and the important and relevant points to the task at hand can all be surmised to a doctor.
We don’t know all the areas we can implement NLP yet but as the world’s data is consistently increasing in quantity (expected to double every two years), we will need to find methods of processing this data in an efficient and error prone manner, NLP could be the solution.
By 2020, it is predicted that 85% of consumer interactions will be handled without a human agent (Chatbots Life, 2019).
67% of US millennials said they are likely to purchase products and services from brands using a chatbot (Chatbots Magazine, 2018).
