Multipart Form Upload using WebClient in .NET Framework 3.5

Prior to my team lead’s resignation, he assigned me a task of updating a legacy codebase written in .NET Framework 3.5. The task itself involved submitting a multipart form request to a Web API, which made it relatively straightforward.

However, despite potential alternatives, my team lead insisted that I continue using .NET Framework 3.5. Furthermore, I was not allowed to incorporate any third-party JSON library into the changes I made.

SHOW ME THE CODE!

The complete source code of this project can be found at https://github.com/goh-chunlin/gcl-boilerplate.csharp/tree/master/console-application/HelloWorld.MultipartForm.

Multipart Form

A multipart form request is a type of form submission used to send data that includes binary or non-textual files, such as images, videos, or documents, along with other form field values. In a multipart form, the data is divided into multiple parts or sections, each representing a different form field or file.

The multipart form data format allows the data to be encoded and organized in a way that can be transmitted over HTTP and processed by web servers. For the challenge given by my team lead, the data consists of two sections, i.e. main_message, a field containing a JSON object, and a list of files.

The JSON object is very simple, which has only two fields called message and documentType, which can only be the value “PERSONAL”, for now.

{
    "message": "...",
    "documentType": "PERSONAL"
}

When the API server handles a multipart form request, it will parse the data by separating the different parts based on the specified boundaries and retrieves the values of each field or file for further processing or storage.

About the Boundaries

In a multipart form, the boundary is a unique string that serves as a delimiter between different parts of the form data. It helps in distinguishing one part from another. The following is an example of what a multipart form data request might look like with the boundary specified.

POST /submit-form HTTP/1.1
Host: example.com
Content-Type: multipart/form-data; boundary=--------------------------8db792beb8632a9

----------------------------8db792beb8632a9
Content-Disposition: form-data; name="main_message"

{
    "message": "...",
    "documentType": "PERSONAL"
}
----------------------------8db792beb8632a9
Content-Disposition: form-data; name="attachments"; filename="picture.jpg"
Content-Type: application/octet-stream

In the example above, the boundary is set to ----------------------------8db792beb8632a9. Each part of the form data is separated by this boundary, as indicated by the dashed lines. When processing the multipart form data on the server, the server uses the boundary to identify the different parts and extract the values of each form field or file.

The format of the boundary string used in a multipart form data request is specified in the HTTP/1.1 specification. The boundary string must meet the following requirements:

  1. It must start with two leading dashes “–“.
  2. It may be followed by any combination of characters, excluding ASCII control characters and the characters used to specify the end boundary (typically dashes).
  3. It should be unique and not appear in the actual data being transmitted.
  4. It should not exceed a length of 70 characters (including the leading dashes).

Besides the leading dashes, the number of dashes depends on how many we want there. It also can be zero, if you like, it is just that more dashes makes the boundary more obvious. Also to make it unique, we use timestamp converted into hexadecimal.

Boundary = "----------------------------" + DateTime.Now.Ticks.ToString("x");

Non-Binary Fields

In the example above, besides sending the files to the server, we also have to send the data in JSON. Hece, we will be having a code to generate the section in following format.

----------------------------8db792beb8632a9
Content-Disposition: form-data; name="main_message"

{
    "message": "...",
    "documentType": "PERSONAL"
}

To do so, we have a form data template variable defined as follows.

static readonly string FormDataTemplate = "\r\n--{0}\r\nContent-Disposition: form-data; name=\"{1}\";\r\n\r\n{2}";

Binary Fields

For the section containing binary file, we will need a code to generate something as follows.

----------------------------8db792beb8632a9
Content-Disposition: form-data; name="attachments"; filename="picture.jpg"
Content-Type: application/octet-stream

Thus, we have the following variable defined.

static readonly string FileHeaderTemplate = "Content-Disposition: form-data; name=\"{0}\"; filename=\"{1}\"\r\nContent-Type: application/octet-stream\r\n\r\n";

Ending BOUNDARY

The ending boundary is required in a multipart form data request to indicate the completion of the form data transmission. It serves as a signal to the server that there are no more parts to process and that the entire form data has been successfully transmitted.

The ending boundary is constructed in a similar way to the regular boundary but with an additional two trailing dashes “–” at the end to indicate its termination. Hence, we have the following codes.

void WriteTrailer(Stream stream)
{
    byte[] trailer = Encoding.UTF8.GetBytes("\r\n--" + Boundary + "--\r\n");
    stream.Write(trailer, 0, trailer.Length);
}

Stream.CopyTo Method

As you may have noticed in our code, we have a method called CopyTo as shown below.

void CopyTo(Stream source, Stream destination, int bufferSize)
{
    byte[] array = new byte[bufferSize];
    int count;
    while ((count = source.Read(array, 0, array.Length)) != 0)
    {
        destination.Write(array, 0, count);
    }
}

The reason we have this code is because the Stream.CopyTo method, that reads the bytes from the current stream and writes them to another stream, is only introduced in .NET Framework 4.0. Hence, we have to write our own CopyTo method.

JSON Handling

To handle the JSON object, the following items are all not available in a .NET Framework 3.5 project.

Hence, if we are not allowed to use any third-party JSON library such as Json.NET from Newtonsoft, then we can only write our own in C#.

string mainMessage =
    "{ " +
        "\"message\": \"" + ConvertToUnicodeString(message) + "\",  " +
        "\"documentType\": \"PERSONAL\" " +
    "}";

The method ConvertToUnicodeString is to support unicode characters in our JSON.

private static string ConvertToUnicodeString(string text)
{
    StringBuilder sb = new StringBuilder();

    foreach (var c in text)
    {
        sb.Append("\\u" + ((int)c).ToString("X4"));
    }

    return sb.ToString();
}

Wrap-Up

This concludes the challenge my team lead presented to me before his resignation.

He did tell me that migrating the project to the modern .NET framework requires significant resources, including development time, training, and potential infrastructure changes. Hence, he foresaw that with limited budgets, the team could only prioritise other business needs over framework upgrades.

After completing the challenge, my team lead has resigned for quite some time. With his departure in mind, I have written this blog post in the hopes that it may offer assistance to other developers facing similar difficulties in their career.

The complete source code of this sample can be found on my GitHub project: https://github.com/goh-chunlin/gcl-boilerplate.csharp/tree/master/console-application/HelloWorld.MultipartForm.

References

Multi-Container ASP .NET Core Web App with Docker Compose

Previously, we have seen how we could containerise our ASP .NET Core 6.0 web app and manage it with docker commands. However, docker commands are mainly for only one image/container. If our solution has multiple containers, we need to use docker-compose to manage them instead.

docker-compose makes things easier because it encompasses all our parameters and workflow into a configuration file in YAML. In this article, I will share my first experience with docker-compose to build mutli-container environments as well as to manage them with simple docker-compose commands.

To help my learning, I will create a simple online message board where people can login with their GitHub account and post a message on the app.

PROJECT GITHUB REPOSITORY

The complete source code of this project can be found at https://github.com/goh-chunlin/Lunar.MessageWall.

Create Multi-container App

We will start with a solution in Visual Studio with two projects:

  • WebFrontEnd: A public-facing web application with Razor pages;
  • MessageWebAPI: A web API project.

By default, the web API project will have a simple GET method available, as shown in the Swagger UI below.

Default web API project created in Visual Studio will have this WeatherForecast API method available by default.

Now, we can make use of this method as a starting point. Let’s have the our client, WebFrontEnd, to call the API and output the result returned by the API to the web page.

var request = new System.Net.Http.HttpRequestMessage();
request.RequestUri = new Uri("http://messagewebapi/WeatherForecast");

var response = await client.SendAsync(request);

string output = await response.Content.ReadAsStringAsync();

In both projects, we will add Container Orchestrator Support with Linux as the target OS. Once we have the docker-compose YAML file ready, we can directly run our docker compose application by simply pressing F5 in Visual Studio.

The docker-compose YAML file for our solution.

Now, we shall be able to see the website output some random weather data returned by the web API.

Congratulations, we’re running a docker compose application.

Configure Authentication in Web App

Our next step is to allow users to login to our web app first before they can post a message on the app.

It’s usually a good idea not to build our own identity management module because we need to deal with a lot more than just building a form to allow users to create an account and type their credentials. One example will be managing and protecting our user’s personal data and passwords. Instead, we should rely on Identity-as-a-Service solutions such as Azure Active Directory B2C.

Firstly, we will register our web app in our Azure AD B2C tenant.

Normally for first-timers, we will need to create a Azure AD B2C tenant first. However, there may be an error message saying that our subscription is not registered to use namespace ‘Microsoft.AzureActiveDirectory’. If you encounter this issue, you can refer to Adam Storr’s article on how to solve this with Azure CLI.

Once we have our Azure AD B2C tenant ready (which is Lunar in my example here), we can proceed to register our web app, as shown below. For testing purposes, we set the Redirect URI to https://jwt.ms, a Microsoft-owned web application that displays the decoded contents of a token. We will update this Redirect URL in the next section below when we link our web app with Azure AD B2C.

Registering a new app “Lunar Message Wall” under the Lunar Tenant.

Secondly, once our web app is registered, we need to create a client secret, as shown below, for later use.

Secrets enable our web app to identify itself to the authentication service when receiving tokens. In addition, please take note that although certificate is recommended over client secret, currently certificates cannot be used to authenticate against Azure AD B2C.

Adding a new client secret which will expire after 6 months.

Thirdly, since we want to allow user authentication with GitHub, we need to create a GitHub OAuth app first.

The Homepage URL here is a temporary dummy data.

After we have registered the OAuth app on GitHub, we will be provided a client ID and client secret. These two information are needed when we configure GitHub as the social identity provider (IDP) on our Azure AD B2C, as shown below.

Configuring GitHub as an identity provider on Azure AD B2C.

Fourthly, we need to define how users interact with our web app for processes such as sign-up, sign-in, password reset, profile editing, etc. To keep thing simple, here we will be using the predefined user flows.

For simplicity, we allow only GitHub sign-in in our user flow.

We can also choose the attributes we want to collect from the user during sign-up and the claims we want returned in the token.

User attributes and token claims.

After we have created the user flow, we can proceed to test it.

In our example here, GitHub OAuth app will be displayed.

Since we specify in our user flow that we need to collect the user’s GitHub display name, there is a field here for the user to enter the display name.

The testing login page from running the user flow.

Setup the Authentication in Frontend and Web API Projects

Now, we can proceed to add Azure AD B2C authentication to our two ASP.NET Core projects.

We will be using the Microsoft Identity Web library, a set of ASP.NET Core libraries that simplify adding Azure AD B2C authentication and authorization support to our web apps.

dotnet add package Microsoft.Identity.Web

The library configures the authentication pipeline with cookie-based authentication. It takes care of sending and receiving HTTP authentication messages, token validation, claims extraction, etc.

For the frontend project, we will be using the following package to add GUI for the sign-in and an associated controller for web app.

dotnet add package Microsoft.Identity.Web.UI

After this, we need to add the configuration to sign in user with Azure AD B2C in our appsettings.json in both projects (The ClientSecret is not needed for the Web API project).

"AzureAdB2C": {
    "Instance": "https://lunarchunlin.b2clogin.com",
    "ClientId": "...",
    "ClientSecret": "...",
    "Domain": "lunarchunlin.onmicrosoft.com",
    "SignedOutCallbackPath": "/signout/B2C_1_LunarMessageWallSignupSignin",
    "SignUpSignInPolicyId": "B2C_1_LunarMessageWallSignupSignin"
}

We will use the configuration above to add the authentication service in Program.cs of both projects.

With the help of the Microsoft.Identity.Web.UI library, we can also easily build a sign-in button with the following code. Full code of it can be seen at _LoginPartial.cshtml.

<a class="nav-link text-dark" asp-area="MicrosoftIdentity" asp-controller="Account" asp-action="SignIn">Sign in</a>

Now, it is time to update the Redirect URI to the localhost. Thus, we need to make sure our WebFrontEnd container has a permanent host port. To do so, we first specify the ports we want to use in the launchsettings.json of the WebFrontEnd project.

"Docker": {
    ...
    "environmentVariables": {
      "ASPNETCORE_URLS": "https://+:443;http://+:80",
      "ASPNETCORE_HTTPS_PORT": "44360"
    },
    "httpPort": 51803,
    "sslPort": 44360
}

Then in the docker-compose, we will specify the same ports too.

services:
  webfrontend:
    image: ${DOCKER_REGISTRY-}webfrontend
    build:
      context: .
      dockerfile: WebFrontEnd/Dockerfile
    ports:
      - "51803:80"
      - "44360:443"

Finally, we will update the Redirect URI in Azure AD B2C according, as shown below.

Updated the Redirect URI to point to our WebFrontEnd container.

Now, right after we click on the Sign In button on our web app, we will be brought to a GitHub sign-in page, as shown below.

The GitHub sign-in page.

Currently, our Web API has only two methods which have different required scopes declared, as shown below.

[Authorize]
public class UserMessageController : ControllerBase
{
    ...
    [HttpGet]
    [RequiredScope("messages.read")]
    public async Task<IEnumerable<UserMessage>> GetAsync()
    {
        ...
    }

    [HttpPost]
    [RequiredScope("messages.write")]
    public async Task<IEnumerable<UserMessage>> PostAsync(...)
    {
        ...
    }
}

Hence, when the frontend needs to send the GET request to retrieve messages, we will first need to get a valid access token with the correct scope.

string accessToken = await _tokenAcquisition.GetAccessTokenForUserAsync(new[] { "https://lunarchunlin.onmicrosoft.com/message-api/messages.read" });

client.DefaultRequestHeaders.Authorization = new AuthenticationHeaderValue("Bearer", accessToken);

client.DefaultRequestHeaders.Accept.Add(new MediaTypeWithQualityHeaderValue("application/json"));

Database

Since we need to store the messages submitted by the users, we will need a database. Here, we use PostgresSQL, an open-source, standards-compliant, and object-relational database.

To run the PostgresSQL with docker-compose we will update our docker-compose.yml file with the following contents.

services:
  ...
  messagewebapi:
    ...
    depends_on:
     - db

  db:
    container_name: 'postgres'
    image: postgres
    environment:
      POSTGRES_PASSWORD: ...

In our case, only the Web API will interact with the database. Hence, we need to make sure that the db service is started before the messagewebapi. In order to specify this relationship, we will use the depends_on option.

User’s messages can now be stored and listed on the web page.

Next Step

This is just the very beginning of my learning journey of dockerising ASP .NET Core solution. In the future, I shall learn more in this area.

References

Protecting Web API with User Password

identity-server

In my previous post, I shared about the way to connect Android app with IdentityServer4 using AppAuth for Android. However, that way will popup a login page on a web browser on phone when users are trying to login to our app. This may not be what the business people want. Sometimes, they are looking for a customized native login page on the app itself.

To do so, we can continue to make use of IdentityServer4.

IdentityServer4 has a grant which is called Resource Owner Password Grant. It allows a client to send username and password to the token service and get an access token back that represents that user. Generally speaking, it is not really recommended to use the AppAuth way. However, since the mobile app is built by our own team, so using the resource owner password grant is okay.

Identity Server Setup: Adding New API Resource

In this setup, I will be using in-memory configuration.

As a start, I need to introduce a new ApiResource with the following codes in the Startup.cs of our IdentityServer project.

var availableResources = new List<ApiResource>();
...
availableResources.Add(new ApiResource("mobile-app-api", "Mobile App API Main Scope"));
...
services.AddIdentityServer()
    ...
    .AddInMemoryApiResources(availableResources)
    .AddInMemoryClients(new ClientStore(Configuration).GetClients())
    .AddAspNetIdentity<ApplicationUser>();

Identity Server Setup: Defining New Client

As the code above shows, there is a ClientStore that we need to add a new client to with the following codes.

public class ClientStore : IClientStore
{
    ...

    public IEnumerable<Client> GetClients()
    {
        var availableClients = new List<Client>();
        
        ...
        
        availableClients.Add(new Client
        {
            ClientId = "mobile-app-api",
            ClientName = "Mobile App APIs",
            AllowedGrantTypes = GrantTypes.ResourceOwnerPassword,
            ClientSecrets = { new Secret(Configuration["MobileAppApi:ClientSecret"].Sha256()) },
            AllowedScopes = { "mobile-app-api" }
        });

        return availableClients;
    }
}

Configuring Services in Web API

In the Startup.cs of our Web API project, we need to update it as follows.

public void ConfigureServices(IServiceCollection services)
{
    services.AddMvc();

    services.AddAuthorization();

    services.AddAuthentication("Bearer")
    .AddIdentityServerAuthentication(options =>
    {
        options.Authority = "<URL of the identity server>";
        options.RequireHttpsMetadata = true;
        options.ApiName = "mobile-app-api";
    });

    services.Configure<MvcOptions>(options =>
    {
        options.Filters.Add(new RequireHttpsAttribute());
    });
}

Configuring HTTP Request Pipeline in Web API

Besides the step above, we also need to make sure the following one line “app.UseAuthentication()” in the Startup.cs. Without this, we cannot make the authentication and authorization to work in our Web API project.

public void Configure(IApplicationBuilder app, IHostingEnvironment env)
{
    ...
    app.UseAuthentication();
    app.UseMvc();
}

Receiving Username and Password to Return Access Token

We also need to add a new controller to receive username and password which will in return tell the mobile app whether the login of the user is successful or not. If the user is logged in successfully, then an access token will be returned.

[Route("api/[controller]")]
public class AuthenticateController : Controller
{
    ...
    [HttpPost]
    [Route("login")]
    public async Task<ActionResult> Login([FromBody] string userName, string password)
    {
        var disco = await DiscoveryClient.GetAsync("<URL of the identity server>");
        var tokenClient = new TokenClient(disco.TokenEndpoint, "mobile-app-api", Configuration["MobileAppApi:ClientSecret"]);
        var tokenResponse = await tokenClient.RequestResourceOwnerPasswordAsync(userName, password, "mobile-app-api");

        if (tokenResponse.IsError)
        {
            return Unauthorized();
        }

        return new JsonResult(tokenResponse.Json);
    }
    ...
}

Securing our APIs

We can now proceed to protect our Web APIs with [Authorize] attribute. In the code below, I also try to return the available claims via the API. The claims will tell the Web API who is logging in and calling the API now via the mobile app.

[HttpGet]
[Authorize]
public IEnumerable<string> Get()
{
    var claimTypesAndValues = new List<string>();

    foreach (var claim in User.Claims)
    {
        claimTypesAndValues.Add($"{ claim.Type }: { claim.Value }");
    }

    return claimTypesAndValues.ToArray();
}

Conclusion

This project took me two days to find out how to make the authentication works because I misunderstand how IdentityServer4 works in this case. Hence, it is always important to fully understand the things on your hands before working on them.

do-not-give-up.png
Do not give up! (Source: A Good Librarian Like a Good Shepherd)

Reference