Image Based CAPTCHA using Jigsaw Puzzle on Blazor

In this article, I will share about how I deploy an image based CAPTCHA as a Blazor app on Azure Static Web App.

PROJECT GITHUB REPOSITORY

The complete source code of this project can be found at https://github.com/goh-chunlin/Lunar.JigsawPuzzleCaptcha.

Project Motivation

CAPTCHA, which stands for “Completely Automated Public Turing test to tell Computers and Humans Apart”, is a type of challenge-response test used in computing to determine whether or not the user is human. Since the day CAPTCHA was invented by Luis von Ahn’s team at Carnegie Mellon University, it has been a reliable tool in separating machines from humans.

In 2007, Luis von Ahn’s team released a programme known as reCAPTCHA which asked users to decipher hard-to-read texts in order to distinguish between human and bots.

A team led by Prof Gao Haichang from Xidian University realised that, with the development of automated computer vision techniques such as OCR, traditional text-based CAPTHCAs are not considered safe anymore for authentication. During the IEEE conference in 2010, they thus proposed a new way, i.e. using an image based CAPTCHA which involves in solving a jigsaw puzzle. Their experiments and security analysis further proved that human can complete the jigsaw puzzle CAPTCHA verification quickly and accurately which bots rarely can. Hence, jigsaw puzzle CAPTCHA can be a substitution to the text-based CAPTCHA.

In 2019, on CSDN (Chinese Software Developer Network), a developer 不写BUG的瑾大大 shared his implementation of jigsaw puzzle captcha in Java. It’s a very detailed blog post but there is still room for improvement in, for example, documenting the code and naming the variables. Hence, I’d like to take this opportunity to implement this jigsaw puzzle CAPTCHA in .NET 5 with C# and Blazor. I also host the demo web app on Azure Static Web App so that you all can access and play with the CAPTCHA: https://jpc.chunlinprojects.com/.

Jigsaw Puzzle CAPTCHA

In a jigsaw puzzle CAPTCHA, there is usually a jigsaw puzzle with at least one misplaced piece where users need to move to the correct place to complete the puzzle. In my demo, I have only one misplaced piece that needs to be moved.

As shown in the screenshot above, there are two necessary images in the CAPTCHA. One of them is a misplaced piece of the puzzle. Another image is the original image with a shaded area indicating where the misplaced piece should be dragged to. What users need to do is just dragging the slider to move the misplaced piece to the shaded area to complete the jigsaw puzzle within a time limit.

In addition, here the CAPTCHA only needs user to drag the missing piece horizontally. This is not only the popular implementation of the jigsaw puzzle CAPTCHA, but also not too challenging for users to pass the CAPTCHA.

Now, let’s see how we can implement this in C# and later deploy the codes to Azure.

Retrieve the Original Image

The first thing we need to do is getting an image for the puzzle. We can have a collection of images that make good jigsaw puzzle stored in our Azure Blob Storage. After that, each time before generating the jigsaw puzzle, we simply need to fetch all the images from the Blob Storage with the following codes and randomly pick one as the jigsaw puzzle image.

public async Task<List<string>> GetAllImageUrlsAsync() 
{
    var output = new List<string>();

    var container = new BlobContainerClient(_storageConnectionString, _containerName);

    var blobItems = container.GetBlobsAsync();

    await foreach (var blob in blobItems) 
    {
        var blobClient = container.GetBlobClient(blob.Name);
        output.Add(blobClient.Uri.ToString());
    }

    return output;
}

Define the Missing Piece Template

To increase the difficulty of the puzzle, we can have jigsaw pieces with different patterns, such as having tabs appearing on different sides of the pieces. In this demo, I will stick to just one pattern of missing piece, which has tabs on the top and right sides, as shown below.

The tabs are basically two circles with the same radius. Their centers are positioned at the middle point of the rectangle side. Hence, we can now build a 2D matrix for the pixels indicating the missing piece template with 1 means inside of the the piece and 0 means outside of the piece.

In addition, we know the general equation of a circle of radius r at origin (h,k) is as follows.

Hence, if there is a point (i,j) inside the circle above, then the following must be true.

If the point (i,j) is outside of the circle, then the following must be true.

With these information, we can build our missing piece 2D matrix as follows.

private int[,] GetMissingPieceData()
{
    int[,] data = new int[PIECE_WIDTH, PIECE_HEIGHT];

    double c1 = (PIECE_WIDTH - TAB_RADIUS) / 2;
    double c2 = PIECE_HEIGHT / 2;
    double squareOfTabRadius = Math.Pow(TAB_RADIUS, 2);

    double xBegin = PIECE_WIDTH - TAB_RADIUS;
    double yBegin = TAB_RADIUS;

    for (int i = 0; i < PIECE_WIDTH; i++)
    {
        for (int j = 0; j < PIECE_HEIGHT; j++)
        {
            double d1 = Math.Pow(i - c1, 2) + Math.Pow(j, 2);
            double d2 = Math.Pow(i - xBegin, 2) + Math.Pow(j - c2, 2);
            if ((j <= yBegin && d1 < squareOfTabRadius) || (i >= xBegin && d2 > squareOfTabRadius))
            {
                data[i, j] = 0;
            }
            else
            {
                data[i, j] = 1;
            }
        }
    }

    return data;
}

After that, we can determine the border of the missing piece easily too from just the template data above. We then can draw the border of the missing piece for better user experience when we display it on screen.

private int[,] GetMissingPieceBorderData(int[,] d)
{
    int[,] borderData = new int[PIECE_WIDTH, PIECE_HEIGHT];

    for (int i = 0; i < d.GetLength(0); i++)
    {
        for (int j = 0; j < d.GetLength(1); j++)
        {
            if (d[i, j] == 0) continue;

            if (i - 1 < 0 || j - 1 < 0 || i + 1 >= PIECE_WIDTH || j + 1 >= PIECE_HEIGHT) 
            {
                borderData[i, j] = 1;

                continue;
            }

            int sumOfSourrounding = 
                d[i - 1, j - 1] + d[i, j - 1] + d[i + 1, j - 1] + 
                d[i - 1, j] + d[i + 1, j] + 
                d[i - 1, j + 1] + d[i, j + 1] + d[i + 1, j + 1];

            if (sumOfSourrounding != 8) 
            {
                borderData[i, j] = 1;
            }
        }
    }

    return borderData;
}

Define the Shaded Area

Next, we need to tell the user where the missing piece should be dragged to. We will use the template data above and apply it to the original image we get from the Azure Blob Storage.

Due to the shape of the missing piece, the proper area to have the shaded area needs to be in the region highlighted in green colour below. Otherwise, the shaded area will not be shown completely and thus give users a bad user experience. The yellow area is okay too but we don’t allow the shaded area to be there to avoid cases where the missing piece covers the shaded area when the images first load and thus confuses the users.

Random random = new Random();

int x = random.Next(originalImage.Width - 2 * PIECE_WIDTH) + PIECE_WIDTH;
int y = random.Next(originalImage.Height - PIECE_HEIGHT);

Let’s assume the shaded area is at the point (x,y) of the original image, then given the original image in a Bitmap variable called originalImage, we can then have the following code to traverse the area and process the pixels in that area.

...
int[,] missingPiecePattern = GetMissingPieceData();

for (int i = 0; i < PIECE_WIDTH; i++)
{
    for (int j = 0; j < PIECE_HEIGHT; j++)
    {
        int templatePattern = missingPiecePattern[i, j];
        int originalArgb = originalImage.GetPixel(x + i, y + j).ToArgb();

        if (templatePattern == 1)
        {
            ...
            originalImage.SetPixel(x + i, y + j, FilterPixel(originalImage, x + i, y + j));
        }
        else
        {
            missingPiece.SetPixel(i, j, Color.Transparent);
        }
    }
}
...

Now we can perform the image convolution with kernel, a 3×3 convolution matrix, as shown below in the FilterPixel method. Here we will be using Box Blur. A Box Blur is a spatial domain linear filter in which each pixel in the resulting image has a value equal to the average value of its neighboring pixels in the input image. By the Central Limit Theorem, repeated application of a box blur will approximate a Gaussian Blur.

For the kernel, I don’t really follow the official Box Blur kernel or Gaussian Blur kernel. Instead, I dim the generated colour by forcing three pixel to be always black (when i = j). This is to make sure the shaded area is not only blurred but darkened.

private Color FilterPixel(Bitmap img, int x, int y)
{
    const int KERNEL_SIZE = 3;
    int[,] kernel = new int[KERNEL_SIZE, KERNEL_SIZE];

    ...

    int r = 0;
    int g = 0;
    int b = 0;
    int count = KERNEL_SIZE * KERNEL_SIZE;
    for (int i = 0; i < kernel.GetLength(0); i++)
    {
        for (int j = 0; j < kernel.GetLength(1); j++)
        {
            Color c = (i == j) ? Color.Black : Color.FromArgb(kernel[i, j]);
            r += c.R;
            g += c.G;
            b += c.B;
        }
    }

return Color.FromArgb(r / count, g / count, b / count);

What will happen when we are processing pixel without all 8 neighbouring pixels? To handle this, we will take the value of the pixel at the opposite position which is describe in the following diagram.

Since we have two images ready, i.e. an image for the missing piece and another image which shows where the missing piece needs to be, we can convert them into base 64 string and send the string values to the web page.

Now, the next step will be displaying these two images on the Blazor web app.

API on Azure Function

When we publish our Blazor app to Azure Static Web Apps, we are getting fast hosting of our web app and scalable APIs. Azure Static Web Apps is designed to host applications where the API and frontend source code lives on GitHub.

The purpose of API in this project is to retrieve the jigsaw puzzle images and verify user submissions. We don’t need a full server for our API because Azure Static Web Apps hosts our API in Azure Functions. So we need to implement our API as Azure Functions here.

We will have two API methods here. The first one is to retrieve the jigsaw puzzle images, as shown below.

[FunctionName("JigsawPuzzleGet")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "get", Route = "jigsaw-puzzle")] HttpRequest req,
    ILogger log)
{
    log.LogInformation("C# HTTP trigger function processed a request.");

    var availablePuzzleImageUrls = await _puzzleImageService.GetAllImageUrlsAsync();

    var random = new Random();
    string selectedPuzzleImageUrl = availablePuzzleImageUrls[random.Next(availablePuzzleImageUrls.Count)];

    var jigsawPuzzle = _puzzleService.CreateJigsawPuzzle(selectedPuzzleImageUrl);
    _captchaStorageService.Save(jigsawPuzzle);

    return new OkObjectResult(jigsawPuzzle);
}

The Azure Function first retrieve all the images from the Azure Blob Storage and then randomly pick one to use in the jigsaw puzzle generation.

Before it returns the puzzle images back in a jigsawPuzzle object, it also saves it into Azure Table Storage so that later when users submit their answer back, we can have another Azure Function to verify whether the users solve the puzzle correctly.

In the Azure Table Storage, we generate a GUID and then store it together with the location of the shaded area, which is randomly generated, as well as an expiry date and time so that users must solve the puzzle within a limited time.

...
var tableClient = new TableClient(...);

...

var entity = new JigsawPuzzleEntity
{
    PartitionKey = ...,
    RowKey = id,
    Id = id,
    X = x,
    Y = y,
    CreatedAt = createdAt,
    ExpiredAt = expiredAt
};

tableClient.AddEntity(entity);
...

Here, GUID is used as the RowKey of the Table Storage. Hence, later when user submits his/her answer, the GUID will be sent back to the Azure Function to help locate back the corresponding record in the Table Storage.

[FunctionName("JigsawPuzzlePost")]
public async Task<IActionResult> Run(
    [HttpTrigger(AuthorizationLevel.Anonymous, "post", Route = "jigsaw-puzzle")] HttpRequest req,
    ILogger log)
{
    log.LogInformation("C# HTTP trigger function processed a request.");

    var body = await new StreamReader(req.Body).ReadToEndAsync();
    var puzzleSubmission = JsonSerializer.Deserialize<PuzzleSubmissionViewModel>(body, new JsonSerializerOptions { PropertyNamingPolicy = JsonNamingPolicy.CamelCase });

    var correspondingRecord = await _captchaStorageService.LoadAsync(puzzleSubmission.Id);

    ...

    bool isPuzzleSolved = _puzzleService.IsPuzzleSolved(...);

    var response = new Response 
    {
        IsSuccessful = isPuzzleSolved,
        Message = isPuzzleSolved ? "The puzzle is solved" : "Sorry, time runs out or you didn't solve the puzzle"
    };

    return new OkObjectResult(response);
}

Since our API is hosted as Azure Function in Consumption Plan, as shown in the screenshot below, we need to note that our code in the Function will be in the serverless mode, i.e. it effectively scales out to meet whatever load it is seeing and scales down when code isn’t running.

Since the Function is in the serverless mode, we will have the issue of serverless cold start. Hence, there will be a latency that users must wait for their function, i.e. the time period starts from when an event happens to a function starts up until that function completes responding to the event. So more precisely, a cold start is an increase in latency for Functions which haven’t been called recently.

In this project, my friend feedbacked to me that he had encountered at least 15 seconds of latency to have the jigsaw puzzle loaded.

Blazor Frontend

Now we can move on to the frontend.

To show the jigsaw puzzle images when the page is loaded, we have the following code.

protected override async Task OnInitializedAsync()
{
    var jigsawPuzzle = await http.GetFromJsonAsync("api/jigsaw-puzzle");
    id = jigsawPuzzle.Id;
    backgroundImage = "data:image/png;base64, " + jigsawPuzzle.BackgroundImage;
    missingPieceImage = jigsawPuzzle.MissingPieceImage;
    y = jigsawPuzzle.Y;
}

Take note that we don’t only get the two images but also the GUID of the jigsaw puzzle record in the Azure Table Storage so that later we can send back this information to the Azure Function for submission verification.

Here, we only return the y-axis value of the shaded area location because users are only allowed to drag the missing puzzle horizontally as discussed earlier. If you would like to increase the difficulty of the CAPTCHA by allowing users to drag the missing piece vertically as well, you can choose not to return the y-axis value.

We then have the following HTML to display the two images.

<div style="margin: 0 auto; padding-left: @(x)px; padding-top: @(y)px; width: 696px; height: 442px; background-image: url('@backgroundImage'); background-size: contain;">
    <div style="width: 88px; height: 80px; background-image: url('data:image/png;base64, @missingPieceImage');">
            
    </div>
</div>

We also have a slider which is binded to the x variable and a button to submit both the value of the x and the GUID back to the Azure Function.

<div style="margin: 0 auto; width: 696px; text-align: center;">
    <input type="range" min="0" max="608" style="width: 100%;" @bind="@x" @bind:event="oninput" />
    <button type="button" @onclick="@Submit">Submit</button>

</div>

The Submit method is as follows which will feedback to users whether they solve the jigsaw puzzle correctly or not. Here I use a toast library for Blazor done by Chris Sainty, a Microsoft MVP.

private async Task Submit()
{
    var submission = new PuzzleSubmissionViewModel
    {
        Id = id,
        X = x
    };
    
    var response = await http.PostAsJsonAsync("api/jigsaw-puzzle", submission);

    var responseMessage = await response.Content.ReadFromJsonAsync<Response>();

    if (responseMessage.IsSuccessful)
    {
        toastService.ShowSuccess(responseMessage.Message);
    }
    else
    {
        toastService.ShowError(responseMessage.Message);
    }
        
}

Now we can test how our app works!

Testing Locally

Before we can test locally, we need to provide the secrets and relevant settings to access Azure Blob Storage and Table Storage.

We first need to have a file called local.settings.json in the root of the Api project with the following content. (Remember to have “Copy to output directly” set to “copy if newer” for the file)

{
  "IsEncrypted": false,
  "Values": {
    "AzureWebJobsStorage": "",
    "FUNCTIONS_WORKER_RUNTIME": "dotnet",
    "CaptchaStorageEndpoint": "...",
    "CaptchaStorageTableName": "...",
    "CaptchaStorageAccountName": "...",
    "CaptchaStorageAccessKey": "...",
    "ImageBlobStorageConnectionString": "...",
    "ImageBlobContainerName": "..."
  },
  "Host": {
    "LocalHttpPort": 7071,
    "CORS": "*"
  }
}

The CORS setting is necessary as well else our Blazor app cannot access the API when we test the web app locally. We don’t have to worry about CORS when we publish it to Azure Static Web Apps because Azure Static Web Apps will automatically configure the app so that it can communicate with the API on Azure using a reverse proxy.

In addition, please remember to exclude local.settings.json from the source control.

In the Client project, since we are going to run our Api at port 7071, we shall let the Client know too. To do so, we first need to specify the base address for local in the Program.cs of the Client project.

builder.Services.AddScoped(sp => new HttpClient { BaseAddress = new Uri(builder.Configuration["API_Prefix"] ?? builder.HostEnvironment.BaseAddress) });

Then we can specify the value for API_Prefix in the appsettings.Development.json in the wwwroot folder.

{
    "API_Prefix": "http://localhost:7071"
}

Finally, please also set both Api and Client projects as the Startup Project in the Visual Studio.

Deploy to Azure Static Web App

After we have created an Azure Static Web Apps resource and bound it with a GitHub Actions which monitors our GitHub repository, the workflow will automatically build and deploy our app and its API to Azure every time we commit or create pull requests into the watched branch. The steps have been described in my previous blog post about Blazor on Azure Static Web App, so I won’t repeat it here.

Since our API needs to have the information of secrets and connection settings to the Azure Storage, we need to specify them under Application Settings of the Azure Static Web App as well. The values will be accessible by API methods in the Azure Functions.

Yup, that’s all for implementing a jigsaw puzzle CAPTCHA in .NET. Feel free to try it out on my Azure Static Web App and let me know your thoughts about it. Thank you!

References

• JAVA 完整实现滑块拼图验证码;
• Blazor WebAssembly with a C# Azure Functions API Sample Project;
• Publish a Blazor WebAssembly app and .NET API with Azure Static Web Apps;
• JS实现滑动拼图验证;
• Image Manipulation: Filters and Convolutions;
• Convolution Matrix;
• ‘ConfigurationBuilder’ does not contain a definition for ‘AddJsonFile’;
• Populate IConfiguration for unit tests;
• IConfiguration does not contain a definition for GetValue;
• Dependency injection, inject with parameters;
• Azure Functions Environment Variables;
• Call a web API from ASP.NET Core Blazor;
• Blazored/Toast;
• Understanding Serverless Cold Start.

The code of this Blazor project described in this article can be found in my GitHub repository: https://github.com/goh-chunlin/Lunar.JigsawPuzzleCaptcha.