Better Trainer

From the middle of November I was a bit more busy with projects @ work so in that time I simply have no time to write any new post, but I have learned a lot, so now I have a lot more material in my head for my blog 🙂

In that months me and my coworker and MVP for any kind of tables – Gašper Kamenšek @ExcelUnplugged – prepared something special for Thrive conference #ThriveConf @ Rimske Terme in November.

Topic of our session talks about how to become better trainer. The idea was to use face detection with emotional scores to see how visitors react and feel in session time. Because feeling is in a correlation with spoken words we use speach recognition. We can connect emotions from faces and words from speach within specific time interval -> so we need to add timestamp to both information. We use Bing Speech API for English language recognition and Google Speech API for Slovenian language recognition.

Because we get from speech recognition API complete sentence, we want to remove some unnecessary words like is, an, the, you etc from it. I use Standford Parser for that purpose – natural language parser that works out the grammatical structure of sentence.

In the end all go to Gašper Kamenšek’s part of project -> to Power BI for visualization and Azure Machine Learning (AML) for powerful analytics of correlation between women and man responses in hapiness for specific words. But that post include just my part of project which is coding (WPF application).

For Face Recognition I use Face API from Azure Cognitive Services. For free you can send limited number of requests to API endpoint so I have to send frame from video in every 3 seconds.
More about this in one of my previous post: Face Detection & Recognition with Azure Face API

For Speech Recognition I use Bing Speech API from Azure Cognitive Services.
More about this in one of my previous post: Speech Recognition (Microsoft Bing Speech API vs. Google Cloud Speech API)

Than we merge faces with words in specific timestamp in ProcessingToPBI function.

private void ProcessingToPBI()
{
    // Remove faces and words that are too old
    facesInTime.RemoveAll(x => x.TimeStamp < lastTimeCheck);
    wordsInTime.RemoveAll(x => x.TimeStamp < lastTimeCheck);

    // Go through all faces ordered ascending by TimeStamp
    foreach (var faceInTime in facesInTime.OrderBy(x => x.TimeStamp))
    {
        // get timestamp of face
        DateTime date = faceInTime.TimeStamp;

        // find all sentences from last time check to timestamp of current frame
        List<WordsInTime> sentences = wordsInTime.Where(x => x.TimeStamp >= lastTimeCheck && x.TimeStamp <= date).ToList();

        // if we have any sentence that match to current frame with detected faces
        if (sentences.Count > 0)
        {
            List<PBIItem> items = new List<PBIItem>();

            // we call Standford Parser for removing unnecessary words
            RemoveUnusedWords(ref sentences);

            // split sentence to individual words for counts
            List<string> words = new List<string>();
            sentences.ForEach(x => x.Word.Split(' ').ToList().ForEach(y => words.Add(y)));

            // Initialize Power BI JSON object for males
            PBIItem mItem = InitializePBIItem(date);

            // Check if we have any male faces detected and update JSON object
            CreatePBIGenderItem(ref mItem, "Male", faceInTime.CameraResult, words.Count(), words.Distinct().Count());

            // Initialize Power BI JSON object for females
            PBIItem fItem = InitializePBIItem(date);

            // Check if we have any female faces detected and update JSON object
            CreatePBIGenderItem(ref fItem, "Female", faceInTime.CameraResult, words.Count(), words.Distinct().Count());

            // go throught all sentences
            foreach (var sentence in sentences)
            {
                // go throught all words in each sentence
                foreach (var word in sentence.Word.Split(' '))
                {
                    if (word.Trim().Length > 0)
                    {
                        // if any male is detected -> create Power BI item for any word
                        if (mItem != null)
                        {
                            PBIItem mItemTemp = (PBIItem)mItem.Clone();

                            mItemTemp.Beseda = word;
                            mItemTemp.Language = sentence.Language;

                            // Add it to items collection for Power BI
                            items.Add(mItemTemp);
                        }

                        // if any female is detected -> create Power BI item for any word
                        if (fItem != null)
                        {
                            PBIItem fItemTemp = (PBIItem)fItem.Clone();

                            fItemTemp.Beseda = word;
                            fItemTemp.Language = sentence.Language;

                            // Add it to items collection for Power BI
                            items.Add(fItemTemp);
                        }
                    }
                }
            }

            // If we have any items for Power BI
            if (items.Count > 0)
            {
                // write it down to log window
                items.ForEach(x => WriteLine_PBI(x.Timestamp + " | Anger: " + x.Anger + " | Contempt: " + x.Contempt + " | Disgust: " + x.Disgust + " | Fear: " + x.Fear + " | Happiness: " + x.Happiness + " | Neutral: " + x.Neutral + " | Sadness: " + x.Sadness + " | Surprise: " + x.Surprise + " | " + x.Beseda + " | " + x.Gender));

                // serialize JSON object with specific Date Formatter
                string json = JsonConvert.SerializeObject(items, Formatting.None, new JsonSerializerSettings
                {
                    DateFormatString = "yyyy-MM-ddTHH:mm:ssZ"
                });

                // If we want to send it to Power BI
                if (sendToPBI)
                {
                    // Create Web Request
                    HttpWebRequest request = (HttpWebRequest)WebRequest.Create(tbAPIURL.Text);
                    request.ContentType = "application/json; charset=utf-8";
                    request.Method = "POST";

                    using (var streamWriter = new StreamWriter(request.GetRequestStream()))
                    {
                        streamWriter.Write(json);
                        streamWriter.Flush();
                        streamWriter.Close();
                    }

                    // Check Web Response
                    WebResponse response = request.GetResponse();
                    var streamReader = new StreamReader(response.GetResponseStream());
                    var result = streamReader.ReadToEnd();

                    try
                    {
                        var wResp = (HttpWebResponse)response;
                        WriteLine_PBI(wResp.StatusCode.ToString());
                    }
                    catch (WebException we)
                    {
                        WriteLine_PBI(((HttpWebResponse)we.Response).StatusCode.ToString());
                    }
                }

                // If we want to send recognized datas to Azure Machine Learning
                if (sendToAML)
                    SendToAML(items);
            }

            // Set timestamp of current frame to last time check
            lastTimeCheck = date;
        }
    }
}

We remove unused words from sentences within RemoveUnusedWords function which use Standford Parser.

Install-Package Stanford.NLP.Parser

After that we send all datas to PowerBI API in JSON format as you can see in ProcessingToPBI function above.

ProcessingToPBI function is triggered every 5 seconds within DispatcherTimer.

timer = new DispatcherTimer();
timer.Tick += Timer_Tick;
timer.Interval = new TimeSpan(0, 0, timerInterval); // timerInterval == 5
timer.Start();

[ Complete code on GitHub ]

Cheers!
Gašper Rupnik

{End.}