Recently I’ve spent some time working on a .NET microservice ecosystem. After working on this for a few weeks, it struck me how time-consuming and frustrating it was to work with the configuration across these services.
It got me thinking about the past projects I’ve worked on and how I’ve never really seen configuration managed well;
it’s always an afterthought. Often a .env file is added when an engineer realises they need to manage configuration
values. Why isn’t it something that’s thought about initially? As applications and teams grow, it can become a critical
pain point.
I often see most projects in an organisation, even a team, managing configuration differently. Sometimes there are
various tools, variables stored in .env files, hard coded and stored in version control, or an external tool (like a
configuration management system.)
This problem has inspired me to spend some time prototyping what good configuration management should look like in a .NET application. I’ve chosen .NET here to demonstrate my findings, as this is the space I’ve been working in recently. However, this approach could easily be applied to any other ecosystem.
Prefer to see the code?
You can find the full example here: https://github.com/rickihastings/dotnet-aws-configuration.
I’ll use an external configuration management system here, specifically AWS Parameter Store, as I have a lot of experience with AWS. However, there is little difference between the various options in the prominent cloud vendors; you could even look at using something like Vault here instead.
To demonstrate this, I’ve built a small application with one endpoint that returns a list of universities from universities.hipolabs.com. The application has three environments, Development, Staging and Production. The actual endpoint lives in the configuration and changes in each environment. I don’t think this is useful in any way, but its purpose is to demonstrate changing URLs between environments.
I’ve recently seen an app settings file per environment, application, and service. The example I’ve witnessed resulted in something like 20+ app settings files, and almost all of them are the same. Some of the values in the files are templates for a pipeline step that replaces tokens with pipeline values. At the same time, not a poor way of doing this (certainly not in other ecosystems anyway); it’s not necessary for .NET.
This practice isn’t specific to this project. I’ve seen this in many places; usually, in the JavaScript ecosystem,
the .local.env and .test.env files have the same problems.
What are the goals I’m hoping to achieve?
The solution should be able to onboard engineers easily, which means no pasting .env files to each other, or looking
through cloud consoles or various other tools to compile a list of configurations.
There should be no hard-coded configuration. I want to react to invalid configuration changes fast; remember, config isn’t always just database URLs. It might be tuning settings for a job processor or how many results an API endpoint returns per page by default. I’ve seen projects have hundreds of these values before.
There are arguments for storing things you don’t want people to change in version control (Cognito user pool IDs, for example). But, I want to scale these up rapidly in production, without going through a code change and passing it through many environments. This solution can often take a while due to end-to-end tests running, which isn’t suitable for me.
My argument against this is don’t always assume you’ll never need to change it. I’ve seen a recent deployment to a new AWS region be a disaster and take much longer than it should have because of hard-coded configurations. Ultimately, this project cost tens of thousands more than it should have, with the root cause pointing to poor configuration management across hundreds of microservices. My other argument against this is that you can still prevent people from updating these values by implementing appropriate authorisation policies in AWS Parameter Store via IAM or whatever other tool you use.
There should be a small number of environment variables hard-coded into pipelines (perhaps tuning variables, JVM,
NODE_OPTIONS, that sort of thing). Running the project against local development should retrieve the configuration
the same way (if possible) as when applications are deployed to your infrastructure.
The solution should validate configuration values as early as possible; this means no runtime errors when a value which should be a number is null.
There should be a solution for serverless applications as well.
A possible solution for containers or services
Fortunately, the configuration ecosystem in modern .NET is excellent and has come a long way since the old Web. config days. It’s possible to overwrite app settings values simply by using environment variables. If I weren’t using an external system for configuration - I’d be doing this.
However, some research led me to discover that AWS has built a library that integrates AWS Parameter Store into .NET’s configuration system.
I’ve decided to create a small class library to attach to any project I use. The library configures the Systems Manager service alongside some other valuable services that the Microsoft configuration ecosystem provides.
public static class ConfigureServices
{
public static IServiceCollection ConfigureSettings<TConfig>(this IServiceCollection services, string key)
where TConfig : class, new()
{
if (services == null)
{
throw new ArgumentNullException(nameof(services));
}
services.AddOptions<TConfig>()
.BindConfiguration(key)
.ValidateDataAnnotations()
.ValidateOnStart();
return services;
}
public static IServiceCollection AddCloudConfiguration(this IServiceCollection services, IConfigurationBuilder builder, string path)
{
builder.AddSystemsManager(path);
return services;
}
}The above class allows me to utilise the Options pattern.
I’m pairing the Options pattern with Microsoft.Extensions.Options.ConfigurationExtensions and Microsoft.Extensions.Options.DataAnnotations.
This combination allows me to describe and validate my application’s possible options. I’ve included the options with
the class library. If you were using this in many different applications, you’d want most of the options in the
consuming service - as they’re most likely to be different across services.
The other benefit of this approach is utilising DataAnnotations for model validation. Data Annotations is a
battle-tested system used heavily in ASP.NET MVC applications, dating back many years. It has a considerable number of
validations and is easily extendable. As you’ll see from the example, I can validate this URL without writing any custom
logic.
public class UniversityServiceOptions
{
public const string Key = "UniversityService";
[Required, Url]
public string Endpoint { get; set; }
}Using this in my Web layer is as simple as doing the following:
// Configuration
builder.Services.AddDefaultAWSOptions(builder.Configuration.GetAWSOptions());
builder.Services.AddCloudConfiguration(builder.Configuration, $"/CloudManagedConfiguration/{builder.Environment.EnvironmentName}/");
// Load required configuration
builder.Services.ConfigureSettings<UniversityServiceOptions>(UniversityServiceOptions.Key);This solution allows me to load the values for this application and the specific environment. If I want to load another list (e.g. a list of generic settings), I can duplicate it and adjust the path. The path will match the values stored in AWS Parameter Store.
Starting the application will fetch the values from AWS, allowing us to update various configuration settings without
making source code changes or broadcasting messages to an entire team or organisation asking people to update their
local .env files. Remember that this solution has a cost implication; Parameter Store isn’t free, although it’s pretty
cheap overall.
As per the pricing example in the AWS documentation;
Cost of 3,000 parameters stored for less than one month = 3,000 * 0.001008 (monthly rate prorated for 15 hours) = $3.02
Cost of 4,000 parameters stored for one month = 4000 * $0.05 = $200
Cost of 7,000 advanced parameters = $200 + $3.02 = $203.02
Cost of 20M API interactions = 20M * $0.05 per 10,000 interactions = $100
Total monthly cost = $203.02 + $100 = $303.021
Thousands of parameters and millions of monthly interactions only cost a few hundred dollars a month—in the grand scheme of things, nothing compared to the cost of your engineers and the time you’ll save them.
What about serverless?
The developer experience in serverless is hit-and-miss. Some parts are excellent, and some things are just annoying and limited. Serverless is where things can get a little bit tricky.
It’s possible to use this solution with serverless; however, reaching out to Parameter Store on every cold start isn’t ideal - it will slow things down, which is unacceptable for me.
Fortunately, AWS can deploy Lambda functions with environment variables automatically populated from Parameter Store. I’m curious if this solution will transfer to other Clouds - although Google and Azure might have similar features, I’ve yet to do the research.
You might have noticed that I abstracted the configuration loading into a new project. Part of the reason for this was;
- Usability between entirely separate services. I would put this into a NuGet package.
- By having multiple ways of loading configurations baked into the class library, I can choose to call the
AddServerlessConfigurationOptionsfunction rather than theAddCloudConfigurationfunction, which will register and validate the options but not load them from AWS Parameter Store.
This solution has two parts; the first part is running functions locally. I want to have the same experience running the traditional application. However, I would also like the same experience when deploying.
In my prototype, I’m using the SAM framework, which is just an extension of CloudFormation. These values work well when deployed, as I can use the dynamic reference to resolve the values. However, frustratingly, this only works when running locally. Another option could be to use the Serverless framework, which does support this, meaning I can use the same solution for both local development and deployed functions.
In my example repo, I’ve created a small script to fetch the environment variables, then run the function with the saved environment variables. I’m still deciding whether I would employ this over the Serverless framework, but I’ve included this to provide multiple options for the different frameworks.
#!/bin/bash
set -e
echo "Fetching environment variables from AWS Parameter Store..."
function="${1:-"Function"}";
environment="${2:-"Development"}";
json=$(aws ssm get-parameters-by-path --path "/CloudManagedConfiguration/$environment/" --recursive | jq -r --arg FUNCTION "$function" '.Parameters[] | .Name = (.Name | split("/") | .[3:length] | join("__")) | {($FUNCTION): {(.Name): .Value}}')
mkdir -p temp
echo $json > temp/environment-variables.json
echo "Completed fetch... Delete ./temp or run ./scripts/fetch-environment-variables to re-fetch."The downside with the SAM and Serverless frameworks is that I have to declare every environment variable I need rather than import all of them based on the path (like the traditional service solution.)
AWSTemplateFormatVersion: '2010-09-09'
Transform: AWS::Serverless-2016-10-31
Description: >
Sample SAM Template for HelloWorld
Parameters:
Environment:
Type: String
Default: Development
AllowedValues: [Development, Staging, Production]
# More info about Globals: https://github.com/awslabs/serverless-application-model/blob/master/docs/globals.rst
Globals:
Function:
Timeout: 10
Resources:
Function:
# More info about Function Resource: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#awsserverlessfunction
Type: AWS::Serverless::Function
Properties:
CodeUri: ./
Handler: CloudManagedConfiguration.Lambda::CloudManagedConfiguration.Lambda.Entrypoint::FunctionHandler
Runtime: dotnet6
Architectures:
- x86_64
MemorySize: 256
# More info about Env Vars: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#environment-object
Environment:
Variables:
ASPNETCORE_ENVIRONMENT: !Ref Environment
UniversityService__Endpoint: !Sub '{{resolve:ssm:/CloudManagedConfiguration/${Environment}/UniversityService/Endpoint}}'
Events:
Index:
# More info about API Event Source: https://github.com/awslabs/serverless-application-model/blob/master/versions/2016-10-31.md#api
Type: Api
Properties:
Path: /
Method: get
Outputs:
# ServerlessRestApi is an implicit API created out of Events key under Serverless::Function
# Find out more about other implicit resources you can reference within SAM
# https://github.com/awslabs/serverless-application-model/blob/master/docs/internals/generated_resources.rst#api
FunctionApi:
Description: "API Gateway endpoint URL for Prod stage for Hello World function"
Value: !Sub "https://${ServerlessRestApi}.execute-api.${AWS::Region}.amazonaws.com/Prod/hello/"
Function:
Description: "Hello World Lambda Function ARN"
Value: !GetAtt Function.Arn
FunctionIamRole:
Description: "Implicit IAM Role created for Hello World function"
Value: !GetAtt FunctionRole.ArnConclusion
By combining a few techniques available in ASP.NET Core, I can create a scalable solution that will allow me to manage configuration across both serverless and traditional applications without worrying about modifying code in version control when configuration changes are required.
Hopefully, this article inspires you to think more about the pain points of managing configuration when scaling up your codebases.