A Kubernetes Lab for Massively Parallel .NET Parameter Sweeps

Let’s start with a problem that many of us in the systems engineering world have faced. You have a computationally intensive application such as a financial model, a scientific process, or in my case, a Discrete Event Simulation (DES). The code is correct, but it is slow.

In some DES problems, to get a statistically reliable answer, you cannot just run it once. You need to run it 5,000 times with different inputs, which is a massive parameter sweep combined with a Monte Carlo experiment to average out the randomness.

If you run this on your developer machine, it will finish in 2026. If you rent a single massive VM on cloud, you are burning money while one CPU core works and the others idle.

This is a brute-force computation problem. How do you solve it without rewriting your entire app? You build a simulation lab on Kubernetes. Here is the blueprint.

About Time

My specific app is a DES built with a C# library called SNA. In DES, the integrity of the entire system depends on a single, unified virtual clock and a centralised Future Event List (FEL). The core promise of the simulation engine is to process events one by one, in strict chronological order.

The FEL is a core component of a DES, which manages and schedules all future events that will occur in the simulation.

This creates an architectural barrier. You cannot simply chop a single simulation into pieces and run them on different pods on Kubernetes. Each pod has its own system clock, and network latency would destroy the causal chain of events. A single simulation run is, by its nature, an inherently single-threaded process.

We cannot parallelise the simulation, but we can parallelise the experiment.

This is what is known as an Embarrassingly Parallel problem. Since the multiple simulation runs do not need to talk to each other, we do not need a complex distributed system. We need an army of independent workers.

The Blueprint: The Simulation Lab

To solve this, I moved away from the idea of a “server” and toward the idea of a “lab”.

Our architecture has three components:

  • The Engine: A containerised .NET app that can run one full simulation and write its results as structured logs;
  • The Orchestrator: A system to manage the parameter sweep, scheduling thousands of simulation pods and ensuring they all run with unique inputs;
  • The Observatory: A centralised place to collect and analyse the structured results from the entire army of pods.

The Engine: Headless .NET

The foundation is a .NET console programme.

We use System.CommandLine to create a strict contract between the container and the orchestrator. We expose key variables of the simulation as CLI arguments, for example, arrival rates, resource counts, service times.

using System.CommandLine;

var rootCommand = new RootCommand
{
Description = "Discrete Event Simulation Demo CLI\n\n" +
"Use 'demo <subcommand> --help' to view options for a specific demo.\n\n" +
"Examples:\n" +
" dotnet DemoApp.dll demo simple-generator\n" +
" dotnet DemoApp.dll demo mmck --servers 3 --capacity 10 --arrival-secs 2.5"
};

// Show help when run with no arguments
if (args.Length == 0)
{
Console.WriteLine("No command provided. Showing help:\n");
rootCommand.Invoke("-h"); // Show help
return 1;
}

// ---- Demo: simple-server ----
var meanArrivalSecondsOption = new Option<double>(
name: "--arrival-secs",
description: "Mean arrival time in seconds.",
getDefaultValue: () => 5.0
);

var simpleServerCommand = new Command("simple-server", "Run the SimpleServerAndGenerator demo");
simpleServerCommand.AddOption(meanArrivalSecondsOption);

simpleServerCommand.SetHandler((double meanArrivalSeconds) =>
{
Console.WriteLine($"====== Running SimpleServerAndGenerator (Mean Arrival (Unit: second)={meanArrivalSeconds}) ======");
SimpleServerAndGenerator.RunDemo(loggerFactory, meanArrivalSeconds);
}, meanArrivalSecondsOption);

var demoCommand = new Command("demo", "Run a simulation demo");
demoCommand.AddCommand(simpleServerCommand);

rootCommand.AddCommand(demoCommand);

return await rootCommand.InvokeAsync(args);

This console programme is then packaged into a Docker container. That’s it. The engine is complete.

The Orchestrator: Unleashing an Army with Argo Workflows

How do you manage a great number of pods without losing your mind?

My first attempt was using standard Kubernetes Jobs. Kubernetes Jobs are primitive, so they are hard to visualise, and managing retries or dependencies requires writing a lot of fragile bash scripts.

The solution is Argo Workflows.

Argo allows us to define the entire parameter sweep as a single workflow object. The killer feature here is the withItems. Alternative, using withParam loop, we can feed Argo a JSON list of parameter combinations, and it handles the rest: Fan-out, throttling, concurrency control, and retries.

apiVersion: argoproj.io/v1alpha1
kind: Workflow
metadata:
generateName: sna-simple-server-job-
spec:
entrypoint: sna-demo
serviceAccountName: argo-workflow
templates:
- name: sna-demo
steps:
- - name: run-simulation
template: simulation-job
arguments:
parameters:
- name: arrival-secs
value: "{{item}}"
withItems: ["5", "10", "20"]

- name: simulation-job
inputs:
parameters:
- name: arrival-secs
container:
image: chunlindocker/sna-demo:latest
command: ["dotnet", "SimNextgenApp.Demo.dll"]
args: ["demo", "simple-server", "--arrival-secs", "{{inputs.parameters.arrival-secs}}"]

This YAML file is our lab manager. It can also be extended to support scheduling, retries, and parallelism, transforming a complex manual task into a single declarative manifest.

The Argo Workflow UI with the fan-out/parallel nodes using the YAML above.

Instead of managing pods, we are now managing a definition of an experiment.

The Observatory: Finding the Needle in a Thousand Haystacks

With a thousand pods running simultaneously, kubectl logs is useless. You are generating gigabytes of text per minute. If one simulation produces an anomaly, finding it in a text stream is impossible.

We solve this with Structured Logging.

By using Serilog, our .NET Engine does not just write text. Instead, it emits machine-readable events with key-value pairs for our parameters and results. Every log entry contains the input parameters (for example, { "WorkerCount": 5, "ServiceTime": 10 }) attached to the result.

These structured logs are sent directly to a centralised platform like Seq. Now, instead of a thousand messy log streams, we have a single, queryable database of our entire experiment results.

Viewing the structured log on Seq generated with Serilog.

Wrap-Up: A Reusable Pattern

This architecture allows us to treat the Kubernetes not just as a place to host websites, but as a massive, on-demand supercomputer.

By separating the Engine from the Orchestrator and the Observatory, we have taken a problem that was too slow for a single machine and solved it using the native strengths of the Kubernetes. We did not need to rewrite the core C# logic. Instead, we just needed to wrap it in a clean interface and unleash a container army to do the work.

The full source code for the SNA library and the Argo workflow examples can be found on GitHub: https://github.com/gcl-team/SNA

The turnout for my DES session in Taipei confirmed a growing hunger in our industry for proactive, simulation-driven approaches to engineering.

P.S. I presented an early version of this blueprint at the Hello World Developer Conference 2025 in Taipei. The discussions with other engineers there were invaluable in refining these ideas.

Beyond the Cert: In the Age of AI

For the fourth consecutive year, I have renewed my Azure Developer Associate certification. It is a valuable discipline that keeps my knowledge of the Azure ecosystem current and sharp. The performance report I received this year was particularly insightful, highlighting both my strengths in security fundamentals and the expected gaps in platform-specific nuances, given my recent work in AWS.

Objectives

Renewing Azure certification is a hallmark of a professional craftsman because it sharpens our tools, knowing our trade. For a junior or mid-level engineer, this path of structured learning and certification is the non-negotiable foundation of a solid career. It is the path I walked myself. It builds the grammar of our trade.

However, for a senior engineer, for an architect, the game has changed. The world is now saturated with competent craftsmen who know the grammar. In the age of AI-assisted coding and brutal corporate “flattening,” simply knowing the tools is no longer a defensible position. It has become table stakes.

The paradox of the senior cloud software engineer is that the very map that got us here, i.e. the structured curriculum and the certification path, is insufficient to guide us to the next level. The renewal assessment results for Microsoft Certified: Azure Developer Associate I received was a perfect map of the existing territory. However, an architect’s job is not to be a master of the known world. It is to be a cartographer of the unknown. The report correctly identified that I need to master Azure specific trade-offs, like choosing ‘Session’ consistency over ‘Strong’ for low-latency scenarios in CosmosDB. The senior engineer learns that rule. The architect must ask a deeper question: “How can I build a model that predicts the precise cost and P99 latency impact of that trade-off for my specific workload, before I write a single line of code?”

Attending AWS Singapore User Group monthly meetup.

About the Results

Let’s make this concrete by looking at the renewal assessment report itself. It was a gift, not because of the score, but because it is a perfect case study in the difference between the Senior Engineer’s path and the Architect’s.

Where the report suggests mastering Azure Cosmos DB five consistency levels, it is prescribing an act of knowledge consumption. The architect’s impulse is to ask a different question entirely: “How can I quantify the trade-off?” I do not just want to know that Session is faster than Strong. I should know, for a given workload, how much faster, at what dollar cost per million requests, and with what measurable impact on data integrity. The architect’s response is to build a model to turn the vendor’s qualitative best practice into a quantitative, predictive economic decision.

This pattern continues with managed services. The report correctly noted my failure to memorise the specific implementation of Azure Container Apps. The path it offers is to better learn the abstraction. The architect’s path is to become professionally paranoid about abstractions. The question is not “What is Container Apps?” but “Why does this abstraction exist, and what are its hidden costs and failure modes?” The architect’s response is to design experiments or simulations to stress-test the abstraction and discover its true operational boundaries, not just to read its documentation.

DHH has just slain the dragon of Cloud Dependency, the largest, most fearsome dragon in our entire cloud industry. (Twitter Source: DHH)

This is the new mandate for senior engineers in this new world where we keep on listening senior engineers being out of work: We must evolve from being consumers of complexity to being creators of clarity. We must move beyond mastering the vendor’s pre-defined solutions and begin forging our own instruments to see the future.

From Cert to Personal Project

This is why, in parallel to maintaining my certifications, I have embarked on a different kind of professional development. It is a path of deep, first-principles creation. I am building a discrete event simulation engine not as a personal hobby project, but as a way to understand more about the most expensive and unpredictable problems in our industry. My certification proves I can solve problems the “Azure way.” This new work is about discovering the the fundamental truths that govern all cloud platforms.

Certifications are the foundation. They are the bedrock of our shared knowledge. However, they are not the lighthouse. In this new era, we must be both.

AWS + Azure.

Certifications are an essential foundation. They represent the bedrock of our shared professional knowledge and a commitment to maintaining a common standard of excellence. However they are not, by themselves, the final destination.

Therefore, my next major “proof-of-work” will not be another certificate. It will be the first in a series of public, data-driven case studies derived from my personal project.

Ultimately, a certificate proves that we are qualified and contributing members of our professional ecosystem. This next body of work is intended to prove something more than that. We need to actively solve the complex, high-impact problems that challenge our industry. In this new era, demonstrating both our foundational knowledge and our capacity to create new value is no longer an aspiration. Instead, it is the new standard.

Together, we learn better.

The Blueprint Fallacy: A Case for Discrete Event Simulation in Modern Systems Architecture

Greetings from Taipei!

I just spent two days at the Hello World Dev Conference 2025 in Taipei, and beneath the hype around cloud and AI, I observed a single, unifying theme: The industry is desperately building tools to cope with a complexity crisis of its own making.

The agenda was a catalog of modern systems engineering challenges. The most valuable sessions were the โ€œ่ธฉ้›ท็ถ“้ฉ—โ€ (landmine-stepping experiences), which offered hard-won lessons from the front lines.

A 2-day technical conference on AI, Kubernetes, and more!

However, these talks raised a more fundamental question for me. We are getting exceptionally good at building tools to detect and recover from failure but are we getting any better at preventing it?

This post is not a simple translation of a Mandarin-language Taiwan conference. It is my analysis of the patterns I observed. I have grouped the key talks I attended into three areas:

  • Cloud Native Infrastructure;
  • Reshaping Product Management and Engineering Productivity with AI;
  • Deep Dives into Advanced AI Engineering.

Feel free to choose to dive into the section that interests you most.

Session: Smart Pizza and Data Observability

This session was led by Shuhsi (ๆž—ๆจน็†™), a Data Engineering Manager at Micron. Micron needs no introduction, they are a massive player in the semiconductor industry, and their smart manufacturing facilities are a prime example of where data engineering is mission-critical.

Micron in Singapore (Credit: Forbes)

Shuhsi’s talk, “Data Observability by OpenLineage,” started with a simple story he called the “Smart Pizza” anomaly.

He presented a scenario familiar to anyone in a data-intensive environment: A critical dashboard flatlines, and the next three hours are a chaotic hunt to find out why. In his “Smart Pizza” example, the culprit was a silent, upstream schema change.

Smart pizza dashboard anomaly.

His solution, OpenLineage, is a powerful framework for what we would callย digital forensics. It is about building a perfect, queryable map of the crime sceneย afterย the crime has been committed. By creating a clear data lineage, it reduces the “Mean Time to Discovery” from hours of panic to minutes of analysis.

Let’s be clear: This is critical, valuable work. Like OpenTelemetry for applications, OpenLineage brings desperately needed order to the chaos of modern data pipelines.

It is a fundamentally reactive posture. It helps us find the bullet path through the body with incredible speed and precision. However, my main point is that our ultimate goal must be to predict the bullet trajectory before the trigger is pulled. Data lineage minimises downtime. My work with simulation, which will be explained in the next session, aims to prevent it entirely by modelling these complex systems to find the breaking points before they break.

Session: Automating a .NET Discrete Event Simulation on Kubernetes

My talk, โ€œSimulation Lab on Kubernetes: Automating .NET Parameter Sweeps,โ€ addressed the wall that every complex systems analysis eventually hits: Combinatorial explosion.

While the industry is focused on understanding past failures, my session is about building the Discrete Event Simulation (DES) engine that can calculate and prevent future ones.

A restaurant simulation game in Honkai Impact 3rd. (Source:
่ฅฟ็ณ – YouTube)

To make this concrete, I used the analogy of a restaurant owner asking, โ€œShould I add another table or hire another waiter?โ€ The only way to answer this rigorously is to simulate thousands of possible futures. The math becomes brutal, fast: testing 50 different configurations with 100 statistical runs each requiresย 5,000 independent simulations. This is not a task for a single machine; it requires a computational army.

My solution is to treat Kubernetes not as a service host, but as a temporary, on-demand supercomputer. The strategy I presented had three core pillars:

  • Declarative Orchestration:ย The entire 5,000-run DES experiment is defined in a single, clean Argo Workflows manifest, transforming a potential scripting nightmare into a manageable, observable process.
  • Radical Isolation:ย Each DES run is containerised in its own pod, creating a perfectly clean and reproducible experimental environment.
  • Controlled Randomness:ย A robust seeding strategy is implemented to ensure that “random” events in our DES are statistically valid and comparable across the entire distributed system.
The turnout for my DES session confirmed a growing hunger in our industry for proactive, simulation-driven approaches to engineering.

The final takeaway was a strategic re-framing of a tool many of us already use. Kubernetes is more than a platform for web apps. It can also be a general-purpose compute engine capable of solving massive scientific and financial modelling problems. It is time we started using it as such.

Session: AI for BI

Denny’s (็›ฃ่ˆœๅ„€) session on “AI for BI” illustrated a classic pain point: The bottleneck between business users who need data and the IT teams who provide it. The proposed solution was a natural language interface, the FineChatBI, a tool designed to sit on top of existing BI platforms to make querying existing data easier.

Denny is introducing AI for BI.

His core insight was that the tool is the easy part. The real work is in building the “underground root system” which includes the immense challenge of defining metrics, managing permissions, and untangling data semantics. Without this foundation, any AI is doomed to fail.

Getting the underground root system right is important for building AI projects.

This is a crucial step forward in making our organisations more data-driven. However, we must also be clear about what problem is being solved.

This is a system designed to provide perfect, instantaneous answers to the question, “What happened?”

My work, and the next category of even more complex AI, begins where this leaves off. It seeks to answer the far harder question: “What will happen ifโ€ฆ?” Sharpening our view of the past is essential, but the ultimate strategic advantage lies in the ability to accurately simulate the future.

Session: The Impossibility of Modeling Human Productivity

The presented Jugg (ๅЉๅ…†ๆญ) is a well-known agile coach and the organiser of Agile Tour Taiwan 2020. His talk, “An AI-Driven Journey of Agile Product Development – From Inspiration to Delivery,” was a masterclass in moving beyond vanity metrics to understand and truly improve engineering performance.

Jugg started with a graph that every engineering lead knows in their gut. As a company grows over time:

  • Business grow (purple line, up);
  • Software architecture and complexity grow (first blue line, up);
  • The number of developers increases (second blue line, up);
  • Expected R&D productivity should grow (green line, up);
  • But paradoxically, the actual R&D productivity often stagnates or even declines (red line, down).

Jugg provided a perfect analogue for the work I do. He tackled the classic productivity paradox: Why does output stagnate even as teams grow? He correctly diagnosed the problem as a failure of measurement and proposed the SPACE framework as a more holistic model for this incredibly complexย human system.

He was, in essence, trying to answer the same class of question I do: “If we change an input variable (team process), how can we predict the output (productivity)?”

This is where the analogy becomes a powerful contrast. Jugg’s world of human systems is filled with messy, unpredictable variables. His solutions are frameworks and dashboards. They are the best tools we have for a system that resists precise calculation.

This session reinforced my conviction that simulation is the most powerful tool we have for predicting performance in the systems we can actually control: Our code and our infrastructure. We do not have to settle for dashboards that show us the past because we can build models that calculate the future.

Session: Building a Map of “What Is” with GraphRAG

The most technically demanding session came from Nils (ๅЉๅฒฆๅดฑ), a Senior Data Scientist at Cathay Financial Holdings. He presented GraphRAG, a significant evolution beyond the “Naive RAG” most of us use today.

Nils is explaining what a Naive RAG is.

He argued compellingly that simple vector search fails because it ignores relationships. By chunking documents, we destroy the contextual links between concepts. GraphRAG solves this by transforming unstructured data into a structured knowledge graph: a web of nodes (entities) and edges (their relationships).

Enhancing RAG-based application accuracy by constructing and leveraging knowledge graphs (Image Credit: LangChain)

In essence,ย GraphRAG is a sophisticated tool for building a static map of a known world.ย It answers the question, “How are all the pieces in our universe connected right now?” For AI customer service, this is a game-changer, as it provides a rich, interconnected context for every query.

This means our data now has an explicit, queryable structure. So, the LLM gets a much richer, more coherent picture of the situation, allowing it to maintain context over long conversations and answer complex, multi-faceted questions.

This session was a brilliant reminder that all advanced AI is built on a foundation of rigorous data modelling.

However, a map, no matter how detailed, is still just a snapshot. It shows us the layout of the city, but it cannot tell us how the traffic will flow at 5 PM.

This is the critical distinction. GraphRAG creates a model of a system at rest and DES creates a model of a system in motion.ย One shows us the relationships while the other lets us press watch how those relationships evolve and interact over time under stress. GraphRAG is the anatomy chart and simulation is the stress test.

Session: Securing the AI Magic Pocket with LLM Guardrails

Nils from Cathay Financial Holdings returned to the stage for Day 2, and this time he tackled one of the most pressing issues in enterprise AI: Security. His talk “Enterprise-Grade LLM Guardrails and Prompt Hardening” was a masterclass in defensive design for AI systems.

What made the session truly brilliant was his central analogy. As he put it, an LLM is a lot like Doraemon: a super-intelligent, incredibly powerful assistant with a “magic pocket” of capabilities. It can solve almost any problem you give it. But, just like in the cartoon, if you give it vague, malicious, or poorly thought-out instructions, it can cause absolute chaos. For a bank, preventing that chaos is non-negotiable.

Nils grounded the problem in the official OWASP Top 10 for LLM Applications.

There are two lines of defence: Guardrails and Prompt Hardening. The core of the strategy lies in understanding two distinct but complementary approaches:

  • Guardrails (The Fortress): An external firewall of input filters and output validators;
  • Prompt Hardening (The Armour): Internal defences built into the prompt to resist manipulation.

This is an essential framework for any enterprise deploying LLMs. It represents the state-of-the-art in building static defences.

While necessary, this defensive posture raises another important question for a developers:ย How does the fortress behave under a full-scale siege?

A static set of rules can defend against known attack patterns. But what about the unknown unknowns? What about the second-order effects? Specifically:

  • Performance Under Attack:ย What is the latency cost of these five layers of validation when we are hit with 10,000 malicious requests per second? At what point does the defence itself become a denial-of-service vector?
  • Emergent Failures:ย When the system is under load and memory is constrained, does one of these guardrails fail in an unexpected way that creates a new vulnerability?

These are not questions a security checklist can answer. They can only be answered by a dynamic stress test. The X-Teaming Nils mentioned is a step in this direction, but a full-scale DES is the ultimate laboratory.

Neil’s techniques are a static set of rules designed to prevent failure. Simulation is a dynamic engine designed to induce failure in a controlled environment to understand a system true breaking points. He is building the armour while my work with DES is in building the testing grounds to see where that armour will break.

Session: Driving Multi-Task AI with a Flowchart in a Single Prompt

The final and most thought-provoking session was delivered by ๅฐน็›ธๅฟ—, who presented a brilliant hack: Embedding a Mermaid flowchart directly into a prompt to force an LLM to execute a deterministic, multi-step process.

ๅฐน็›ธๅฟ—๏ผŒๆ•ธๆ“šๆฑบ็ญ–่‚กไปฝๆœ‰้™ๅ…ฌๅธๆŠ€่ก“้•ทใ€‚

He provided a new way beyond the chaos of autonomous agents and the rigidity of external orchestrators like LangGraph. By teaching the LLM to read a flowchart, he effectively turns it into a reliable state machine executor. It is a masterful piece of engineering that imposes order on a probabilistic system.

Action Grounding Principles proposed by ็›ธๅฟ—.

What he has created is the perfect blueprint. It is a model of a process as it should run in a world with no friction, no delays, and no resource contention.

And in that, he revealed the final, critical gap in our industry thinking.

A blueprint is not a stress test. A flowchart cannot answer the questions that actually determine the success or failure of a system at scale:

  • What happens when 10,000 users try to execute this flowchart at once and they all hit the same database lock?
  • What is the cascading delay if one step in the flowchart has a 5% chance of timing out?
  • Where are the hidden queues and bottlenecks in this process?

His flowchart is the architect’s beautiful drawing of an airplane. A DES is theย wind tunnel. It is the necessary, brutal encounter with reality that shows us where the blueprint will fail under stress.

The ability to define a process is the beginning. The ability to simulate that process under the chaotic conditions of the real world is the final, necessary step to building systems that don’t just look good on paper, but actually work.

Final Thoughts and Key Takeaways from Taipei

My two days at the Hello World Dev Conference were not a tour of technologies. In fact, they were a confirmation of a dangerous blind spot in our industry.

From what I observe, they build tools for digital forensics to map past failures. They sharpen their tools with AI to perfectly understand what just happened. They create knowledge graphs to model the systems at rest. They design perfect, deterministic blueprints for how AI processes should work.

These are all necessary and brilliant advancements in the art of mapmaking.

However, the critical, missing discipline is the one that asks not “What is the map?”, but “What will happen to the city during the hurricane?” The hard questions of latency under load, failures, and bottlenecks are not found on any of their map.

Our industry is full of brilliant mapmakers. The next frontier belongs to people who can model, simulate, and predict the behaviour of complex systems under stress, before the hurricane reaches.

That is why I am building SNA, my .NET-based Discrete Event Simulation engine.

Hello, Taipei. Taken from the window of the conference venue.

I am leaving Taipei with a notebook full of ideas, a deeper understanding of the challenges and solutions being pioneered by my peers in the Mandarin-speaking tech community, and a renewed sense of excitement for the future we are all building.

Securing APIs with OAuth2 Introspection

In today’s interconnected world, APIs are the backbone of modern apps. Protecting these APIs and ensuring only authorised users access sensitive data is now more crucial than ever. While many authentication and authorisation methods exist, OAuth2 Introspection stands out as a robust and flexible approach. In this post, we will explore what OAuth2 Introspection is, why we should use it, and how to implement it in our .NET apps.

Before we dive into the technical details, let’s remind ourselves why API security is so important. Think about it: APIs often handle the most sensitive stuff. If those APIs are not well protected, we are basically opening the door to some nasty consequences. Data breaches? Yep. Regulatory fines (GDPR, HIPAA, you name it)? Potentially. Not to mention, losing the trust of our users. A secure API shows that we value their data and are committed to keeping it safe. And, of course, it helps prevent the bad guys from exploiting vulnerabilities to steal data or cause all sorts of trouble.

The most common method of securing APIs is using access tokens as proof of authorization. These tokens, typically in the form of JWTs (JSON Web Tokens), are passed by the client to the API with each request. The API then needs a way to validate these tokens to verify that they are legitimate and haven’t been tampered with. This is where OAuth2 Introspection comes in.

OAuth2 Introspection

OAuth2 Introspection is a mechanism for validating bearer tokens in an OAuth2 environment. We can think of it as a secure lookup service for our access tokens. It allows an API to query an auth server, which is also the “issuer” of the token, to determine the validity and attributes of a given token.

The workflow of an OAuth2 Introspection request.

To illustrate the process, the diagram above visualises the flow of an OAuth2 Introspection request. The Client sends the bearer token to the Web API, which then forwards it to the auth server via the introspection endpoint. The auth server validates the token and returns a JSON response, which is then processed by the Web API. Finally, the Web API grants (or denies) access to the requested resource based on the token validity.

Introspection vs. Direct JWT Validation

You might be thinking, “Isn’t this just how we normally validate a JWT token?” Well, yesโ€ฆ and no. What is the difference, and why is there a special term “Introspection” for this?

With direct JWT validation, we essentially check the token ourselves, verifying its signature, expiry, and sometimes audience. Introspection takes a different approach because it involves asking the auth server about the token status. This leads to differences in the pros and cons, which we will explore next.

With OAuth2 Introspection, we gain several key advantages. First, it works with various token formats (JWTs, opaque tokens, etc.) and auth server implementations. Furthermore, because the validation logic resides on the auth server, we get consistency and easier management of token revocation and other security policies. Most importantly, OAuth2 Introspection makes token revocation straightforward (e.g., if a user changes their password or a client is compromised). In contrast, revoking a JWT after it has been issued is significantly more complex.

.NET Implementation

Now, let’s see how to implement OAuth2 Introspection in a .NET Web API using the AddOAuth2Introspection authentication scheme.

The core configuration lives in our Program.cs file, where we set up the authentication and authorisation services.

// ... (previous code for building the app)

builder.Services.AddAuthentication("Bearer")
.AddOAuth2Introspection("Bearer", options =>
{
options.IntrospectionEndpoint = "<Auth server base URL>/connect/introspect";
options.ClientId = "<Client ID>";
options.ClientSecret = "<Client Secret>";

options.DiscoveryPolicy = new IdentityModel.Client.DiscoveryPolicy
{
RequireHttps = false,
};
});

builder.Services.AddAuthorization();

// ... (rest of the Program.cs)

This code above configures the authentication service to use the “Bearer” scheme, which is the standard for bearer tokens. AddOAuth2Introspection(โ€ฆ) is where the magic happens because it adds the OAuth2 Introspection authentication handler by pointing to IntrospectionEndpoint, the URL our API will use to send the token for validation.

Usually, RequireHttps needs to be true in production. However, in situations like when the API and the auth server are both deployed to the same Elastic Container Service (ECS) cluster and they communicate internally within the AWS network, we can set it to false. This is because the Application Load Balancer (ALB) handles the TLS/SSL termination and the internal communication between services happens over HTTP, we can safely disable RequireHttps in the DiscoveryPolicy for the introspection endpoint within the ECS cluster. This simplifies the setup without compromising security, as the communication from the outside world to our ALB is already secured by HTTPS.

Finally, to secure our API endpoints and require authentication, we can simply use the [Authorize] attribute, as demonstrated below.

[ApiController]
[Route("[controller]")]
[Authorize]
public class MyController : ControllerBase
{
[HttpGet("GetData")]
public IActionResult GetData()
{
...
}
}

Wrap-Up

OAuth2 Introspection is a powerful and flexible approach for securing our APIs, providing a centralised way to validate bearer tokens and manage access. By understanding the process, implementing it correctly, and following best practices, we can significantly improve the security posture of our apps and protect our valuable data.

References