Use Azure Cognitive Services to recognize karting images with Custom Vision service

For this we will use the public preview of Custom Vision Service Prediction API.

In the previous article about Custom Vision Service we saw how to train Azure Cognitive Services to recognize karting images thanks to Custom Vision Service Training API in a .NET console application.

Today we will see how to actually recognize karting images using Custom Vision Service Prediction API relying on the model we trained with our own data.

To start we will create a project with the .NET Console App template in Visual Studio.

Before starting you will need to have a Prediction Key and a Training Key. You can get it from www.customvision.ai. If you have an Azure account get it by creating a Custom Vision Service in the Azure Portal or as seen in a previous article via ARM template.

Creation

The goal here is to build a console application that will allow us to predict karting and other racing sports images using a previously trained Custom Vision Service project.

The first thing you will need is to add the following NuGet packages to your project: Microsoft.Cognitive.CustomVision.Prediction and Microsoft.Cognitive.CustomVision.Training

Let's create the main method of our console application and a start method:

using Microsoft.Cognitive.CustomVision.Prediction;
using Microsoft.Cognitive.CustomVision.Training;
...
using Microsoft.Rest;
using System;
using System.Linq;
using System.Threading.Tasks;
...

namespace VisionRacing.PredictingRacingImages
{
    class Program
    {
        static void Main(string[] args)
        {
            var trainingKey = "Your Custom Vision training key.";
            var predictionKey = "Your Custom Vision prediction key.";

            Start(trainingKey, predictionKey).Wait();
        }

        private static async Task Start(string trainingKey, string predictionKey)
        {
            var projectName = " ";
            var trainingApi = GetTrainingApi(trainingKey);
            var predictionEndpoint = GetPredictionEndpoint(predictionKey);

            while (!string.IsNullOrEmpty(projectName))
            {
                try
                {
                    Console.Clear();
                    await ListProjects(trainingApi);

                    Console.WriteLine("Please enter a project name or press enter to exit:");
                    projectName = Console.ReadLine();

                    if (!string.IsNullOrEmpty(projectName))
                    {
                        await WorkOnProject(trainingApi, predictionEndpoint, projectName);
                    }
                }
                catch (Exception ex)
                {
                    Console.ForegroundColor = ConsoleColor.Red;
                    Console.WriteLine($"An error occurred: {Environment.NewLine}{ex.Message}");

                    if (ex is HttpOperationException)
                    {
                        Console.WriteLine(((HttpOperationException)ex).Response.Content);
                    }

                    Console.ResetColor();
                    Console.WriteLine();
                    Console.WriteLine();
                    Console.WriteLine("Press any key to continue");
                    Console.ReadLine();
                }
            }
        }

        ...

        private static PredictionEndpoint GetPredictionEndpoint(string predictionKey)
        {
            return new PredictionEndpoint
            {
                ApiKey = predictionKey
            };
        }

        private static TrainingApi GetTrainingApi(string trainingKey)
        {
            return new TrainingApi
            {
                ApiKey = trainingKey
            };
        }

        private static async Task ListProjects(TrainingApi trainingApi)
        {
            var projects = await trainingApi.GetProjectsAsync();

            if (projects.Any())
            {
                Console.WriteLine($"Existing projects: {Environment.NewLine}{string.Join(Environment.NewLine, projects.Select(p => p.Name))}{Environment.NewLine}");
            }
        }

        ...
    }
}

The start method will allow us to list all the projects associated to our account. You should enter the name of an existing trained project.

We also create two other methods:

GetTrainingApi: this will return a TrainingApi object using the provided Training Key.
GetPredictionEndpoint: this will return a PredictionEndpoint object using the provided Prediction Key.
ListProjects: it will call the TrainingApi method GetProjectsAsync to list all the projects.

When a project name is entered, the WorkOnProject method is called:

using Microsoft.Cognitive.CustomVision.Prediction;
using Microsoft.Cognitive.CustomVision.Training;
using Microsoft.Cognitive.CustomVision.Training.Models;
...
using System;
using System.Linq;
using System.Threading.Tasks;
...

namespace VisionRacing.PredictingRacingImages
{
    class Program
    {
        ...
        private static async Task WorkOnProject(TrainingApi trainingApi, PredictionEndpoint predictionEndpoint, string name)
        {
            var option = " ";

            while (!string.IsNullOrEmpty(option))
            {
                Console.Clear();

                var project = await GetOrCreateProject(trainingApi, name);
                Console.WriteLine($"  --- Project {project.Name} ---");
                Console.WriteLine();

                await ListProjectTags(trainingApi, project.Id);

                Console.WriteLine("Type an option number:");
                Console.WriteLine("  1: Predict Karting images");
                Console.WriteLine("  2: Predict F1 images");
                Console.WriteLine("  3: Predict MotoGP images");
                Console.WriteLine("  4: Predict Rally images");
                Console.WriteLine("  5: Predict Test images");
                Console.WriteLine();
                Console.WriteLine($"Press any other key to exit project {name}");
                option = Console.ReadLine();

                switch (option)
                {
                    case "1":
                        await StartPrediction(predictionEndpoint, project.Id, ImageType.Karting);
                        break;
                    case "2":
                        await StartPrediction(predictionEndpoint, project.Id, ImageType.F1);
                        break;
                    case "3":
                        await StartPrediction(predictionEndpoint, project.Id, ImageType.MotoGP);
                        break;
                    case "4":
                        await StartPrediction(predictionEndpoint, project.Id, ImageType.Rally);
                        break;
                    case "5":
                        await StartPrediction(predictionEndpoint, project.Id, ImageType.Test);
                        break;
                    default:
                        option = string.Empty;
                        break;
                }
            }
        }
        
        ...

        private static async Task<Project> GetOrCreateProject(TrainingApi trainingApi, string name)
        {
            var projects = await trainingApi.GetProjectsAsync();

            var project = projects.Where(p => p.Name.ToUpper() == name.ToUpper()).SingleOrDefault();

            if (project == null)
            {
                project = await trainingApi.CreateProjectAsync(name);
            }

            return project;
        }

        ...

        private static async Task ListProjectTags(TrainingApi trainingApi, Guid projectId)
        {
            var tagList = await trainingApi.GetTagsAsync(projectId);

            if (tagList.Tags.Any())
            {
                Console.WriteLine($"Tags: {Environment.NewLine}{string.Join(Environment.NewLine, tagList.Tags.Select(t => $"  {t.Name} (Image count: {t.ImageCount})"))}{Environment.NewLine}");
            }
            else
            {
                Console.WriteLine($"No tags yet...{Environment.NewLine}");
            }
        }

        ...

        private enum ImageType
        {
            F1,
            Karting,
            MotoGP,
            Rally,
            Test
        }
    }
}

We will also use two other methods:

GetOrCreateProject: will try to retrieve an existing project. Create it if doesn't exists.
ListProjectTags: list all the existing tags of a project.

The workOnProject method will get a project using the name provided, list the project tags and then show several options.

The five different options will allow to predict different kind of images: Karting, F1, Moto GP, Rally and Test images.

Here is the code that shows how to predict a type of image:

using Microsoft.Cognitive.CustomVision.Prediction;
...
using System;
using System.Linq;
using System.Threading.Tasks;
using ImageUrl = Microsoft.Cognitive.CustomVision.Prediction.Models.ImageUrl;

namespace VisionRacing.PredictingRacingImages
{
    class Program
    {
        ...
        private static int GetImageCountPerImageType(ImageType imageType)
        {
            switch (imageType)
            {
                case ImageType.F1:
                    return 7;
                case ImageType.Karting:
                    return 35;
                case ImageType.MotoGP:
                    return 7;
                case ImageType.Rally:
                    return 6;
                case ImageType.Test:
                    return 10;
                default:
                    return 0;
            }
        }

        private static string GetImageDescriptionPerImageTypeAndNumber(ImageType imageType, int imageNumber)
        {
            switch (imageType)
            {
                case ImageType.Test:
                    switch (imageNumber)
                    {
                        case 1:
                        case 2:
                            return "Solo kart racer on track";
                        case 3:
                        case 7:
                        case 10:
                            return "Multiple kart racers on track";
                        case 4:
                        case 9:
                            return "Solo kart racer on pre-grid";
                        case 5:
                            return "Kart racers on a podium";
                        case 6:
                            return "A tent in a karting paddock";
                        case 8:
                            return "A racing helmet";
                        default:
                            return string.Empty;
                    }
                case ImageType.F1:
                case ImageType.Karting:
                case ImageType.MotoGP:
                case ImageType.Rally:
                default:
                    return string.Empty;
            }
        }

        ...

        private static async Task StartPrediction(PredictionEndpoint predictionEndpoint, Guid projectId, ImageType imageType)
        {
            var imageTypeCount = GetImageCountPerImageType(imageType);

            for (int i = 1; i <= imageTypeCount; i++)
            {
                Console.Clear();
                Console.WriteLine($"Image {imageType} {i}/{imageTypeCount} prediction in progress...");

                var imageDescription = GetImageDescriptionPerImageTypeAndNumber(imageType, i);

                if (!string.IsNullOrEmpty(imageDescription))
                {
                    Console.WriteLine();
                    Console.WriteLine($"Description: {imageDescription}");
                }

                var imagePredictionResult = await predictionEndpoint.PredictImageUrlWithNoStoreAsync(projectId, new ImageUrl($"https://github.com/vivienchevallier/Article-AzureCognitive.Vision-Racing/raw/master/Images/{imageType}/{imageType}%20({i}).jpg"));

                Console.Clear();

                if (imagePredictionResult.Predictions.Any())
                {
                    Console.WriteLine($"Image {imageType} {i}/{imageTypeCount}: {imageDescription}{Environment.NewLine}{string.Join(Environment.NewLine, imagePredictionResult.Predictions.Select(p => $"  {p.Tag}: {p.Probability:P1}"))}{Environment.NewLine}");
                }
                else
                {
                    Console.WriteLine($"Image {imageType} {i}/{imageTypeCount}: no predictions yet...{Environment.NewLine}");
                }

                if (i < imageTypeCount)
                {
                    Console.WriteLine("Press enter to predict next image or any other key to stop predictions");

                    if (Console.ReadKey().Key != ConsoleKey.Enter)
                    {
                        break;
                    }
                }
                else
                {
                    Console.WriteLine("All images predicted... Press any key to continue");
                    Console.ReadLine();
                }
            }
        }
        ...
    }
}

Here are more details about the methods used:

GetImageCountPerImageType: get the number of images on GitHub for the specified ImageType.
GetImageDescriptionPerImageTypeAndNumber: providing an image type and its number, return a description of the image.
StartPrediction: predict all the images of an image type by sending a request to the service for each image.

Example of use

The console application is now ready to run, let's execute it:

Existing projects:
Test Project

Please enter a project name or press enter to exit:

When starting it will list the existing projects and allow to enter a project name, new or existing. If you followed the previous article steps, your trained project should appear in the list.

  --- Project Test Project ---


Tags:
  Karting (Image count: 35)
  Racing (Image count: 35)

Type an option number:
  1: Predict Karting images
  2: Predict F1 images
  3: Predict MotoGP images
  4: Predict Rally images
  5: Predict Test images


Press any other key to exit project Test Project

Select option five to try to predict the karting images that are in the test images.

Image Test 1/10 prediction in progress...

Description: Solo kart racer on track

...

Image Test 1/10: Solo kart racer on track
  Racing: 99.9%
  Karting: 99.9%

Press enter to predict next image or any other key to stop predictions

The first image is a solo kart racer on track and is unknown from our previously trained project. The service is able to predict with a probability of 99% that this image can be recognized with tags Racing and Karting, amazing!

The same goes with image 2, images 3, 7 and 10 showing multiple kart racers on track, images 4 and 9 showing a solo kart racer on pre-grid.

However, image 5 showing kart racers on a podium and image 6 showing a tent in a karting paddock are predicted with a probability of 0%. Image 8 showing a racing helmet gets the same result.

Considering that only 35 images were used to train our project we can say that the results are really good.

One all the test images are predicted you'll get the following output:

Image Test 10/10: Multiple kart racers on track
  Racing: 99.9%
  Karting: 99.9%

All images predicted... Press any key to continue

To go further

Now that you have predicted test images you could try to predict one of the four other types like F1.

You are then going to discover unexpected results. Karting and Formula 1 being very close, the service will sometimes predict an image with a Formula 1 as a karting image.

Image F1 1/7:
  Karting: 15.1%
  Racing: 13.9%

Press enter to predict next image or any other key to stop predictions

First F1 image is not predicted as a karting image, but images from 2 to 7 are with a probability close to 100%.

If it happens, you'll have to train your project with new images. In our case we went to add F1 images and then train our project to refine it.

Once that done using the console application built in the previous article, here the results we get:

Image F1 1/7:
  F1: 99.6%
  Racing: 99.5%
  Karting: 0.0%

Press enter to predict next image or any other key to stop predictions

Image F1 2/7:
  Racing: 100.0%
  F1: 99.9%
  Karting: 0.2%

Press enter to predict next image or any other key to stop predictions

This is pretty impressive! We trained our project with only seven new images with the tags Racing and F1 and we are able to predict that all seven F1 images can be recognized with the tags Racing and F1 with a probability of almost 100%. But also, the probability of those images to be recognized with the tag Karting is close to 0%.

Then imagine how accurate a project can be when trained with thousands of images and multiples tags!

Summary

In this article and the previous one, we have built two NET console applications allowing us to train Azure Cognitive Services to recognize racing/karting images with Custom Vision Service Training API and accurately predict racing/karting images with Custom Vision Service Prediction API.

Now don't forget that the Custom Vision service is still in preview, new features like object detection are added, APIs and SDKs will be updated.

Custom Vision Service Prediction is a really interesting and powerful service, I'll definitely keep an eye on it to see how it evolves!

You can get the project source code here:

Browse the GitHub repository

(Note that the project uses .NET Framework 4.7)

Please feel free to comment or contact me if you have any question about this article.

Add a comment

Language

Name

E-mail (Will not be published)

Website url

Message

Back to articles