Storing ASP.NET Core Data Protection Keys in the Database

If you use Pocket, you will no doubt know the pain of articles stacking up faster than you can read them… The ReadMore chrome extension was developed to help solve this problem by showing you a random article that you have previously saved to Pocket.

The extension allows users to view an article and then delete or archive that article. The API exposes endpoints for carrying out all of these actions. As the user had specifically given us the ability to modify their articles, one of the considerations was to not leak this privilege. In particular, the API should only be able to delete or archive articles that it had returned in a response.

Data Protection APIs

While looking for ways to solve this problem, I came across the ASP.NET Core Data Protection APIs. They provide the ability to easily encrypt and decrypt data, allowing that data to be safely exposed out with the application. Using these APIs allows an article’s Pocket ID to be returned in an encrypted form. When it is received by the endpoints for deleting or archiving an article, the ID is then decrypted and used. This satisfies the requirement that only articles returned by the API can be deleted or archived.

However, there was one issue to overcome — key storage. The Data Protection APIs, by default, handle the auto-generation of keys for carrying out the encryption and decryption. The behaviour of the APIs can be modified by registering implementations of specific interfaces with the service provider. For instance, an implementation of IXmlRepository can be registered to change how keys are stored.

The Data Protection APIs come with several implementations of the IXmlRepository interface. Unfortunately, as the API for the extension is hosted on Heroku, none of these options were particularly useful. Ideally, I wanted to store the keys in the database. So, I set out to create a custom IXmlRepository implementation.

Implementing IXmlRepository

The actual implementation used in the API for the extension can be found here, but for this post I’ve cut things down a bit. The IXmlRepository interface consists of two methods:

• GetAllElements()— responsible for returning a collection containing all the stored elements
• StoreElement(XElement element, string friendlyName) — handles the storing of an element

To store the keys, we just need a simple table to hold the XML for each key:

CREATE TABLE "XmlKeys"(
   Id UUID PRIMARY KEY     NOT NULL default gen_random_uuid(),
   Xml             TEXT    NOT NULL
);

Based on that table, we can implement IXmlRepository:

// Based on the implementation in https://github.com/defining-technology/read-more-api.
// Dapper is used to help with executing SQL queries, but you could use any
// abstraction over IDbConnection that you prefer.
public class SqlXmlRepository : IXmlRepository
{
    private const string TableName = "\"XmlKeys\"";
    private readonly Func<IDbConnection> _connectionFactory;

    public SqlXmlRepository(Func<IDbConnection> connectionFactory)
    {
        _connectionFactory = connectionFactory;
    }

    public IReadOnlyCollection<XElement> GetAllElements()
    {
        using (var conn = Connection)
        {
            return conn.Query<XmlKey>($"select * from {TableName}")
                .Select(x => XElement.Parse(x.Xml))
                .ToList();
        }
    }

    public void StoreElement(XElement element, string friendlyName)
    {
        var key = new XmlKey
        {
            Xml = element.ToString(SaveOptions.DisableFormatting)
        };
        using (var conn = Connection)
        {
            return conn.ExecuteScalar<Guid>($"INSERT into {TableName}(Xml) values (@Xml) returning Id", key);
        }
    }

    private IDbConnection Connection
    {
        get
        {
            var conn = _connectionFactory();
            conn.Open();
            return conn;
        }
    }
}

Overall, there’s not too much to it — it uses an IDbConnection instance to execute INSERT and SELECT queries, while also handling converting the XML to a string and back again.

Run Migrations Before Creating Keys

The move from ASP.NET Core 1.x to 2.x isn’t all plain sailing. In 1.x, an initial key was generated when it was first needed. In 2.x, this was changed to generate that initial key when the application started. The outcome of this change was that the tables did not exist when this initial key was created, as the migration scripts had not yet been run.

After searching around the implementation for the Data Protection APIs (thank goodness for open source!), I found out that an implementation of IStartupFilter was added by the Data Protection APIs to handle this key creation. This explained why the key was being created before the migrations were being run — all registered IStartupFilter instances are processed prior to Startup.Configure being called, which was the method handling the execution of the migration scripts (special thanks goes to this great article on IStartupFilters).

The answer was to implement an IStartupFilter instance to handle the migrations (the actual implementation used can be found here).

Wiring it All Together

The last step in all this is to register the instances.

First off, we register our instance of IXmlRepository. Once we’ve done that, we register our IStartupFilter instance so that it executes before the IStartupFilter registered by the Data Protection APIs. Finally, we can call AddDataProtection() to enable Data Protection.

public class Startup
{
    // ...

    public void ConfigureServices(IServiceCollection services)
    {
        services.AddMvc();

        // ...

        // this part expects that you've already registered an IDbConnection implemention
        services.AddSingleton<Func<IDbConnection>>(provider => provider.GetService<IDbConnection>);
        services.AddSingleton<IXmlRepository, SqlXmlRepository>();

        // Add a startup filter to handle migrations here, so that the required
        // tables are present before the KeyManager attempts to retrieve data
        // protection keys via the SqlXmlRepository
        services.TryAddEnumerable(ServiceDescriptor.Singleton<IStartupFilter, DbMigrationsStartupFilter>());

        var sp = services.BuildServiceProvider();
        services.AddDataProtection()
            .AddKeyManagementOptions(options => options.XmlRepository = sp.GetService<IXmlRepository>());
    }

    // ...
}

Unfortunately there’s a bit of a quirk between the ASP.NET Core 1.x and 2.x implementations of the Data Protection APIs. According to the docs, registering an implementation should be all that is required to override the default IXmlRepository implementation:

To change the default repository application-wide, register a custom singleton IXmlRepository in the service provider

While this worked in the 1.x version, it does not work in the 2.x version. So, at least for now, we need to add the line found after the call to AddDataProtection above, to explicitly set the XmlRepository instance to be used.

Wrap Up

I hope this helps anyone else looking at how to store Data Protection keys in a database, if you’ve got any questions or comments, just respond to this article or drop me a tweet.