At Trailhead, we have built many enterprise-grade .NET APIs that deal with managing documents – files, photos, PDFs, etc – and in this blog I will distill some best practices that will help you optimize your API and download performance, reducing the processing burden and memory requirements for your API. These techniques work equally well in both Amazon AWS and Microsoft Azure.

The Big Picture – Your Challenge as an Enterprise Architect

Imagine we are building an application, where some of the domain objects in the model may reference Documents – these could be photos, contracts, attachments, etc. In this case, let’s say the User object references a profile picture document. You are storing the actual documents as BLOBS (Binary Large Objects) in the Cloud, either Amazon S3 Buckets or Azure Storage Account Containers.

Your database holds some metadata about these documents, for instance which domain objects they belong to or are associated with (in this case the User table has a ProfilePictureId foreign key to the Documents table), and you have a .NET API used by a web client, say an Angular or React SPA. You can use the API to fetch objects, for instance a User, but you also want to fetch or upload the corresponding document – i.e. the profile picture, a fairly large JPEG image. 

You are of course concerned about security – so your BLOB storage is private and you only want to hand out access to specific documents for specific API callers based on your application logic, and only want to allow controlled uploads of new documents.

Finally, you are concerned about performance. You want the web app to be able to leverage the CDN (Content Delivery Network) of the cloud BLOB provider for cached access to documents. You would rather not funnel all the document uploads and downloads through your API, as these files are large, and this would require some carefully crafted streaming mechanisms that don’t require the whole document to be in the API memory as it moves from BLOB storage to the API response. With a potentially large number of web clients and simultaneous document operations, you also don’t want your API to become the bottleneck or need to scale up or scale out to handle all these document operations – which could also become a costly option.

This blog lays out an approach that addresses all of these needs and concerns. The diagram below illustrates the key components:

Solution Components

Let’s start by briefly examining the components of this solution:

The Cloud BLOB Provider

This is where the documents are actually stored. In AWS, this is in Buckets, and the files can have path delimiters in their filenames, which create a virtual folder structure. In Azure, they are in Containers within a Storage Account, and again can have a filename with a virtual folder path. In both cases, a file can be uniquely described by its path and filename, which we will call it’s Key. We don’t want to use the original file name as this filename, as it needs to be unique within its folder, and we can’t control the file names selected for upload, so we need to create our own unique filenames.

The Database

The database can be a relational database like Azure Sql Database or AWS RDS, or it could be a documents database like Mongo or Cosmos DB. It contains two key tables:

• Documents contains metadata about each document, like its original file name, size in bytes, content type, a unique ID that it can be referenced by in the database model and its key – the unique pointer to the document in the cloud BLOB provider. The data in this table is sufficient to model relationships between domain objects and documents, and create various APIs that return this metadata – like a list of files (with original filename, sizes and content types) for a given domain object. You create your own business logic to filter and restrict that the API returns based on maybe the user’s Role or some other more fine grained mechanism. 
• Uploads will be used to track the status of ongoing document uploads, and can be seen as a transient table that represents something that once complete will become a Document. It’s very similar in structure to the document table, but before an upload happens, we don’t know some of the data – like its file size and content type. Also, there is no guarantee an upload will complete – it could fail or be aborted. For these reasons, the Uploads are kept separate from Documents, even though similar, and technically they could live in one table with a status column and some nullable allowances. Uploads will contain the document original FileName, a status, and the proposed cloud BLOB key

The Web App

The web app is our enterprise application, typically a SPA (Single Page Application), written in Angular or React. It will have some UI that shows in this case the user profile picture for the logged in user, and allow you to upload a new picture. In the general case, the application might have some document management screens, say showing attachments related to some business object, or contracts for a claim.

Authentication and Authorization

The SPA user is typically authenticated using some OpenIdConnect/Oauth2 IDP (IDentity Provider), which could be cloud native like Azure Active Directory or Amazon Cognito, or it could be deployed along with the API, maybe using Duende Identity Server. The user makes secure calls over HTTPS to the API using an access token issued by the IDP (this access token is usually a JWT – a JSON Web Token), and the token contains claims about the identity and roles of the user making the calls, which the API can use to implement access control logic for documents and other entities.

The API

The API is typically a .NET (dotnet core) REST API, currently using the new .NET Minimal API paradigm. It exposes queries for fetching information about objects and relationships, as well as commands for creating. updating and deleting objects. In this example, you can call /userprofile/me to get information about yourself, including a link to your profile picture, and some other methods to assist with uploading a new profile picture – more on that later.

The Upload Handler

The Upload Handler is an Azure Function or AWS Lambda that is invoked when a file upload to a cloud BLOB provider completes, via an event or notification sent by the BLOB provider.

The Flow

With these components outlined, let’s get into the detailed flow of how this all fits together, starting with the process to upload a new document (profile picture).

Beginning the Upload

Assign a Key: To begin a file upload, the application calls an API /documents/begin-upload, passing the original filename of the document. The API generates a unique Key for this document by combining a random GUID with the original filename. This Key will be where the file will be uploaded to in the cloud BLOB provider. If you want to group different kinds of documents in different folders or containers, you can include this logic in the key generation – the main thing is you assign a unique path and filename to where this document is going to end up landing once uploaded. 

Generate a URL: Once you have the key, both Azure and Amazon have a concept of providing a special kind of URL that has some built in security that will allow a web client (or anything that you give this URL) to perform some allowed operation(s) for some specified amount of time. In Azure they are called Shared Access Signatures, and in AWS they are Presigned URLs. Both allow you to specify allowed operations (like GET, PUT), and a time limit. There are many other nuances – like you can create keys for whole containers of folders and employ all kinds of revocable access control strategies, but for our purposes, all we need to do here is pass the Key, and ask for a say 5 minute URL for uploading (PUT) a file to the location specified by the Key. 

Create an Upload Record: Now that we have the URL, we write that along with the key and original FileName to the Uploads table, along with a status which we initially set to New. At this point there is no Document record, just an Upload record. Note that we can add other information to this Upload record too – like which user created it and when.

Return the Upload ID and the URl: Finally the API will return the ID of the Upload record, as well as the URL, to the caller. The ID will be used later by the caller to check on the upload status, and the URL will be used to actually upload the file.

The File Upload

The web client now has an Upload ID and a URL which can be used to upload the actual document to the cloud BLOB provider. This is just a regular file upload call. Here is one example for Azure (you can simplify it greatly by just setting options.path to the URL returned by begin-upload. There are a couple of gotchas here:

• For Azure, you need so set the x-ms-blob-type header to the value BlockBlob
• For Azure, you need to enable CORS on the Blob Container (allow the Origins and Methods that you want to restrict to, for instance just from your web app and just GET, HEAD, OPTIONS and PUT)
• Send the Content-Length Header to match the actual file size
• For Azure, you can send Content-Type if you know it, or Azure will automatically detect it on upload. For AWS, either send it (in which case you also need to specify that same content type in the pre signed URL generation), or leave it unspecified and you will get a generic application/octet-stream
• Don’t add any additional headers or query parameters in the web app, as the URL returned by the Cloud BLOB Provider for upload has a signature, and if you mess with the URL when performing the upload, the signature will not match and the upload will be rejected. (If you need to attach custom headers sent by the client, you can include those when generating the URL in the API).

If the file upload fails, or takes longer than the time limit in the URL, the Web Client can show an error message and the user can try again.

Tracking the Upload Status

Once the file upload completes, there are two parallel courses of action: on the back end, the upload handler will receive an upload complete event and create the document record – but in the meantime the web application needs to wait for that to happen, and this is done by polling the upload table. To do this, the web application calls the /documents/check-upload-status API, passing the upload ID. This will initially just return the status New, but once the upload handler is done processing, it will either return Failed or Complete – along with a URL that the client can use to download/view the document. More on that URL below.

Completing the Upload

When the document upload to AWS S3 or Azure Storage Accounts completes, you can configure the Cloud Provider to send a notification event to interested parties. There are many ways to skin this cat, below is the simplest setup:

• For Azure, EventGrid provides the event notification process. See this article on how to enable this on your subscription, and then go to your storage account and add a Subscription under Events that routes the event to your Azure function that implements the upload handler.
• For AWS, you can directly trigger a Lambda when an upload completes. Go to your S3 bucks in the AWS console web site and under Properties, Event Notifications, create an Event Notification for the Put event and select your lambda function as the Destination (note you can also route to an SNS Topic or SQS queue, and have your Lambda subscribe to those events instead).

Once configured, create your Azure Function or AWS Lambda to handle the event. Both the AWS SDK and the Azure Functions technology stack are rapidly evolving, so at the time of writing it was quite hard to get accurate information on the best way to write these handlers, so I’ll provider some tips here that maybe will make life easier:

AWS

For AWS, you will want to use these packages:

<PackageReference Include="AWSSDK.Core" Version="3.7.106.41" />
<PackageReference Include="Amazon.Lambda.Core" Version="2.1.0" />
<PackageReference Include="Amazon.Lambda.S3Events" Version="3.0.1" />
<PackageReference Include="Amazon.Lambda.Serialization.SystemTextJson" Version="2.3.1" />

Your handler can then look something like this

using System.Threading.Tasks;
using Amazon.Lambda.Core;
using Amazon.Lambda.S3Events;

[assembly: LambdaSerializer(typeof(Amazon.Lambda.Serialization.SystemTextJson.DefaultLambdaJsonSerializer))]

public class FileUploadHandler : LambdaServiceBase {
    public async Task HandleUpload(S3Event request, ILambdaContext context) {
       foreach (var s3R in request.Records) {
            var s3 = s3R.S3;
            await storageService.CompleteUploadAsync(s3.Bucket.Name, s3.Object.Key, s3.Object.Size);
        }
    }
}

Note you get the File Size, bucket name and key as part of the S3 event which is handy when looking up the Upload record. _storageService does the actual work of looking up the Upload record based on the Key, updating its status to Complete, and creating the Document record (with help of the information in the Upload Record, plus the File Size and Content Type). Unfortunately you don’t get the Content-Type with the S3Event, but you can look it up with this snippet:

await _s3Client.GetObjectMetadataAsync(new GetObjectMetadataRequest { BucketName = bucket, Key = key });

Azure

For your Azure function you will want these packages:

<PackageReference Include="Microsoft.Azure.Functions.Worker" Version="1.14.1" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Sdk" Version="1.10.0" />
<PackageReference Include="Microsoft.Azure.Functions.Worker.Extensions.EventGrid" Version="3.2.1" />

Your function will receive a EventGridData object and can pass it off to your upload logic as follows:

using Microsoft.Azure.Functions.Worker;
using System.Threading.Tasks;

namespace TrailheadTechnology.Azure.Workers.Functions;

public class EventGridProcessor {
    [Function(nameof(FileUploadedAsync))]
    public async Task FileUploadedAsync([EventGridTrigger] EventGridData input, FunctionContext context, ILogger logger) {
        await _storageService.CompleteUploadAsync(
            input.data.url, input.data.contentLength, input.data.contentType);
    }
}

The model for EventGridData is as follows

public class Data
{
    public string api { get; set; }
    public string clientRequestId { get; set; }
    public string requestId { get; set; }
    public string eTag { get; set; }
    public string contentType { get; set; }
    public long contentLength { get; set; }
    public string blobType { get; set; }
    public string url { get; set; }
    public string sequencer { get; set; }
    public StorageDiagnostics storageDiagnostics { get; set; }
}

public class EventGridData
{
    public string topic { get; set; }
    public string subject { get; set; }
    public string eventType { get; set; }
    public DateTime eventTime { get; set; }
    public string id { get; set; }
    public Data data { get; set; }
    public string dataVersion { get; set; }
    public string metadataVersion { get; set; }
}

public class StorageDiagnostics
{
    public string batchId { get; set; }
}

For Azure, you don’t get the ‘Key’ of the Blob in the event – you get the full URL of the Blob. However it’s quite easy to parse the URL and extract the part that corresponds to the Key, so we can use that to look up the Upload record, then set it to Complete, and create our Document record. For Azure, we DO get the Content Type in the event, which Azure detects for us, so we don’t need to make any additional calls0

Note: When creating the Document record, if you stored who created the Upload record in that table, you can also copy that to the Document record, so that you have an audit of the origin of the document.

Viewing the Document

Once the upload handler has done its work and written a document record, the document can be viewed. The downloading of documents is much simpler, and also leverages Shared Access Signatures or in AWS Presigned URLs. 

To generate a URL for an existing document, look up the Key in the Documents table, and then just use the same method you used during the Document Upload flow to request a Presigned URL or shared access key. However, this time generate it for GET instead of PUT, and make the lifetime longer, maybe 30 minutes instead of 5. This URL can then be returned to the caller, and the Web application can bind to the URL in some UI element, which will directly download from the Cloud BLOB Provider to the browser. The signature on the URL is used by the Cloud BLOB Provider to verify access, but the actual content can come from a CDN endpoint close to the browser.

If the UI is such that you have say a list of attachments for some object, and the user may not actually click on the attachment until much later, you may want to have the client call your API to get the URL for the attachment on demand, as the user clicks it, rather than when the links are first retrieved – otherwise they may expire.

Finally – going back to the “Tracking the Upload Status” for the web application – the API call /documents/check-upload-status uses similar logic to return a URL for download. If the Uploads table for the specified ID has a status of Complete, then the corresponding Document is looked up using the Key in the Document table, and then a Presigned Url/Shared Access Signature URL for GET is generated and returned along with the Upload ID and Status.

Conclusion

There are many pieces to this puzzle, and I did omit some of the details in order to keep the blog somewhat manageable. I tried to include details where I had to do a lot of research myself, and skipped the easier parts.

With this architecture pattern, you can limit file uploads and downloads to be an operation solely handled by the web application and the Cloud BLOB Provider, taking the API out of the loop, except for some minimal metadata, and thus greatly increase performance and scalability, while reducing cost, without sacrificing control and security.

I hope you find this pattern useful!