Dos and Don’ts for Streaming File Uploads to Azure Blob Storage With .NET MVC

What’s the big deal about file uploads? Well, the big deal is that they are tricky operations. Implement file uploads the wrong way, and you may end up with memory leaks, server slowdowns, out-of-memory errors, and worst of all, unhappy users. 

With Azure Blob Storage, there are multiple different ways to implement file uploads. But if you want to let your users upload large files, you will almost certainly want to do it using streams.  

You’ll find a lot of file upload examples out there that use what I call the “small file” methods (such as IFormFile) or a byte array, a memory stream buffer, etc. These are fine for small files, but I wouldn’t recommend them for file sizes over 2MB. For larger file size situations, we need to be much more careful about how we process the file. 

What NOT To Do 

Here are some of the “don’ts” for .NET MVC for uploading large files to Azure Blob Storage: 

DON’T Do It if You Don’t Have To 

You may be able to use client-side direct uploads if your architecture supports generating SAS (Shared Access Signature) upload Uris, and if you don’t need to process the upload through your API. Handling large file uploads is complex and you should see if you can offload that functionality to Azure Blob Storage entirely before tackling it. 

DON’T Use IFormFile for Large Files 

If you let MVC try to bind to an IFormFile, it will attempt to spool the entire file into memory. That is exactly what we don’t want to do with large files. 

DON’T Model Bind at All, in Fact 

MVC is very good at model binding from the web request. But when it comes to files, any sort of model binding will try to, you guessed it, read the entire file into memory. This is slow and wasteful if all we want to do is forward the data right on to Azure Blob Storage. 

DON’T Use Any Memory Streams 

This one should be kind of obvious because what does a memory stream do? Yes, read the file into memory. For the same reasons as above, we don’t want to do this. 

DON’T Use a Byte Array Either 

Yep, same reason. Your byte array will work fine for small files or light loading, but how long will you have to wait to put that large file into that byte array? And if there are multiple files? Just don’t do it. There is a better way! 

So, What Are the DOs? 

There is one example in Microsoft’s documentation that covers this topic very well for .NET MVC, and it is here, in the last section about large files. In fact, if you are reading this article, I highly recommend you read that entire document (and the related example) because it covers the “large file vs small file” differences and has a lot of great information. And just go ahead and download the whole example, because it has some of the pieces we need.  

At the time of this article, the latest version of the sample code available is for .NET Core 3.0 but the pieces we need will work just fine with .NET 7. 

The other piece we need is getting the file to Azure Blob Storage during the upload process. To do that, we are going to use several of the helpers and guidance from the MVC example on file uploads. Here are the important parts: 

DO Use a Multipart Form-Data Request 

You’ll see this in the file upload example. Multipart (multipart/form-data) requests are a special type of request designed for sending streams that can also support sending multiple files or pieces of data. I think the explanation in the swagger documentation is also really helpful to understand this type of request. 

The multipart request (which can actually be for a single file) can be read with a MultipartReader that does NOT need to spool the body of the request into memory. By using the multipart form-data request you can also support sending additional data through the request. 

 

It is important to note that although “multipart” is in the name, the multipart request does not mean that a single file will be sent in parts.

This is not the same as file “chunking”, although the name sounds similar. Chunking files is a separate technique for file uploads – and if you need some features such as the ability to pause and restart or retry partial uploads, chunking may be the way to go. 

DO Prevent MVC From Model-Binding the Request 

The example linked above has an attribute class that works perfectly for this: DisableFormValueModelBindingAttribute.cs. With it, we can disable the model binding on the Controller Action that we want to use. 

DO Increase or Disable the Request Size Limitation 

This depends on your requirements. You can set the size to something reasonable, depending on the file sizes you want to allow. If you get larger than 256MB (the current max for single block upload for blob storage), you may need to do the streaming setup described here, and ALSO chunk the files across blobs. (Be sure to read the most current documentation to ensure your file sizes are supported with the method you choose.) 

/// <summary> 
/// Upload an document using our streaming method 
/// </summary> 
/// <returns>A collection of document models</returns> [DisableFormValueModelBinding] [ProducesResponseType(typeof(List<DocumentModel>), 200)] [DisableRequestSizeLimit] [HttpPost("streamupload")] public async Task<IActionResult> UploadDocumentStream() ...

DO Process the Boundaries of the Request and Send the Stream to Azure Blob Storage 

Again, this comes mostly from Microsoft’s example, with some special processing to copy the stream of the request body for a single file to Azure Blob Storage.  

The file content type can be read without touching the stream, along with the filename. But remember, neither of these can always be trusted. You should encode the filename and, if you really want to prevent unauthorized types, you could go even further by adding some checking to read the first few bytes of the stream and verify the type. 

var sectionFileName = contentDisposition.FileName.Value; 
// use an encoded filename in case there is anything weird 
var encodedFileName = WebUtility.HtmlEncode(Path.GetFileName(sectionFileName)); 
// now make it unique 
var uniqueFileName = $"{Guid.NewGuid()}_{encodedFileName}"; 
// read the section filename to get the content type 
var fileContentType = MimeTypeHelper.GetMimeType(sectionFileName); 
// check the mime type against our list of allowed types 
var enumerable = allowedTypes.ToList(); 
if (!enumerable.Contains(fileContentType.ToLower())) 
{
    return new ResultModel<List<DocumentModel>>("fileType", "File type not allowed: " + fileContentType); 
} 

DO Look at the Final Position of the Stream To Get the File Size 

If you want to get or save the file size, you can check the position of the stream after uploading it to blob storage. Do this instead of trying to get the length of the stream beforehand. 

DO Remove Any Signing Key From the Uri if You Are Preventing Direct Downloads 

The Uri that is generated as part of the blob will include an access token at the end. If you don’t want to let your users have direct blob access, you can trim this part off. 

// trick to get the size without reading the stream in memory 
var size = section.Body.Position; 
// check size limit in case somehow a larger file got through. we can't do it until after the upload because we don't want to put the stream in memory 
if (maxBytes < size) 
{ 
    await blobClient.DeleteIfExistsAsync(); 
    return new ResultModel<List<DocumentModel>>("fileSize", "File too large: " + encodedFileName); 
} 
var doc = new DocumentModel() 
{
    FileName = encodedFileName, 
    MimeType = fileContentType, 
    FileSize = size, 
// Do NOT include Uri query since it has the SAS credentials; This will return the URL without the querystring. 
// UrlDecode to convert %2F into "/" since Azure Storage returns it encoded. This prevents the folder from being included in the filename. 
    Url = WebUtility.UrlDecode(blobClient.Uri.GetLeftPart(UriPartial.Path)) 
}; 

DO Use a Stream Upload Method to Blob Storage 

There are multiple upload methods available, but make sure you choose one that has an input of a Stream, and use the section.Body stream to send the upload. 

var blobClient = blobContainerClient.GetBlobClient(uniqueFileName); 
// use a CloudBlockBlob because both BlobBlockClient and BlobClient buffer into memory for uploads 
CloudBlockBlob blob = new CloudBlockBlob(blobClient.Uri); 
await blob.UploadFromStreamAsync(section.Body); 
// set the type after the upload, otherwise will get an error that blob does not exist 
await blobClient.SetHttpHeadersAsync(new BlobHttpHeaders { ContentType = fileContentType }); 

DO Performance-Profile Your Results 

This may be the most important instruction! 

 

After you’ve written your code, run it in Release mode using the Visual Studio Performance Profiling tools. Compare your profiling results to that of a known memory-eating method, such as an IFormFile.

Be aware that different versions of the Azure Blob Storage library may perform differently. And different implementations may also perform differently! Here were some of my results: 

To do this simple profiling, I used PostMan to upload multiple files of around 20MB in several requests. By using a collection, or by opening multiple tabs, you can submit multiple requests at a time to see how the memory of the application is consumed. 

First, using an IFormFile. You can see the memory usage increases rapidly for each request with this method. 

Next, using the latest version (v12) of the Azure Blob Storage libraries and a Stream upload method. Notice that it’s not much better than IFormFile!  

Although BlobStorageClient is the latest way to interact with blob storage, when I look at the memory snapshots of this operation it has internal buffers (at least, at the time of this writing) that cause it to not perform too well when used in this way. 

var blobClient = blobContainerClient.GetBlobClient(uniqueFileName); 
await blobClient.UploadAsync(section.Body); 

But we can see a much better memory performance by using almost identical code and the previous library version that utilizes CloudBlockBlob instead of BlobClient. The same file uploads result in a small increase (due to resource consumption that eventually goes back down with garbage collection), but nothing near the ~600MB consumption like above. I’m sure whatever memory issues exist with the latest libraries will be resolved eventually, but I will use this method for now. 

// use a CloudBlockBlob because both BlobBlockClient and BlobClient buffer into memory for uploads 
CloudBlockBlob blob = new CloudBlockBlob(blobClient.Uri); 
await blob.UploadFromStreamAsync(section.Body); 

For your reference, here is a version of the upload service methods from that last profiling result: 

/// <summary> 
/// Upload multipart content from a request body 
/// </summary> 
/// <param name="requestBody">body stream from the request</param> 
/// <param name="contentType">content type from the request</param> 
/// <returns></returns> 
public async Task<ResultModel<List<DocumentModel>>> UploadMultipartDocumentRequest(Stream requestBody, string contentType) 
{ 
  // configuration values hardcoded here for testing 
  var bytes = 104857600; 
  var types = new List<string>{ "application/pdf", "image/jpeg", "image/png"}; 
  var docs = await this.UploadMultipartContent(requestBody, contentType, types, bytes); 
  if (docs.Success) 
  { 
    foreach (var doc in docs.Result) 
    { 
      // here we could save the document data to a database for tracking 
      if (doc?.Url != null) 
      { 
        Debug.WriteLine($"Document saved: {doc.Url}"); 
      } 
    } 
  } 
  return docs; 
} 
/// <summary> 
/// Upload multipart content from a request body 
/// based on microsoft example https://github.com/dotnet/AspNetCore.Docs/tree/main/aspnetcore/mvc/models/file-uploads/samples/ 
/// and large file streaming example https://docs.microsoft.com/en-us/aspnet/core/mvc/models/file-uploads?view=aspnetcore-5.0#upload-large-files-with-streaming 
/// can accept multiple files in multipart stream 
/// </summary> 
/// <param name="requestBody">the stream from the request body</param> 
/// <param name="contentType">content type from the request</param> 
/// <param name="allowedTypes">list of allowed file types</param> 
/// <param name="maxBytes">max bytes allowed</param> 
/// <returns>a collection of document models</returns> 
public async Task<ResultModel<List<DocumentModel>>> UploadMultipartContent(Stream requestBody, string contentType, List<string> allowedTypes, int maxBytes) 
{ 
  // Check if HttpRequest (Form Data) is a Multipart Content Type 
  if (!IsMultipartContentType(contentType)) 
  { 
    return new ResultModel<List<DocumentModel>>("requestType", $"Expected a multipart request, but got {contentType}"); 
  } 
  FormOptions defaultFormOptions = new FormOptions(); 
  // Create a Collection of KeyValue Pairs. 
  var formAccumulator = new KeyValueAccumulator(); 
  // Determine the Multipart Boundary. 
  var boundary = GetBoundary(MediaTypeHeaderValue.Parse(contentType), defaultFormOptions.MultipartBoundaryLengthLimit); 
  var reader = new MultipartReader(boundary, requestBody); 
  var section = await reader.ReadNextSectionAsync(); 
  List<DocumentModel> docList = new List<DocumentModel>(); 
  var blobContainerClient = GetBlobContainerClient(); 
  // Loop through each 'Section', starting with the current 'Section'. 
  while (section != null) 
  { 
    // Check if the current 'Section' has a ContentDispositionHeader. 
    var hasContentDispositionHeader = ContentDispositionHeaderValue.TryParse(section.ContentDisposition, out ContentDispositionHeaderValue contentDisposition); 
    if (hasContentDispositionHeader) 
    { 
      if (HasFileContentDisposition(contentDisposition)) 
      { 
        try 
        { 
          var sectionFileName = contentDisposition.FileName.Value; 
          // use an encoded filename in case there is anything weird 
          var encodedFileName = WebUtility.HtmlEncode(Path.GetFileName(sectionFileName)); 
          // now make it unique 
          var uniqueFileName = $"{Guid.NewGuid()}_{encodedFileName}"; 
          // read the section filename to get the content type 
          var fileContentType = MimeTypeHelper.GetMimeType(sectionFileName); 
          // check the mime type against our list of allowed types 
          var enumerable = allowedTypes.ToList(); 
          if (!enumerable.Contains(fileContentType.ToLower())) 
          { 
            return new ResultModel<List<DocumentModel>>("fileType", "File type not allowed: " + fileContentType); 
          } 
          var blobClient = blobContainerClient.GetBlobClient(uniqueFileName); 
          // use a CloudBlockBlob because both BlobBlockClient and BlobClient buffer into memory for uploads 
          CloudBlockBlob blob = new CloudBlockBlob(blobClient.Uri); 
          await blob.UploadFromStreamAsync(section.Body); 
          // set the type after the upload, otherwise will get an error that blob does not exist 
          await blobClient.SetHttpHeadersAsync(new BlobHttpHeaders { ContentType = fileContentType }); 
          // trick to get the size without reading the stream in memory 
          var size = section.Body.Position; 
          // check size limit in case somehow a larger file got through. we can't do it until after the upload because we don't want to put the stream in memory 
          if (maxBytes < size) 
          { 
            await blobClient.DeleteIfExistsAsync(); 
            return new ResultModel<List<DocumentModel>>("fileSize", "File too large: " + encodedFileName); 
          } 
          var doc = new DocumentModel() 
          { 
            FileName = encodedFileName, 
            MimeType = fileContentType, 
            FileSize = size, 
            // Do NOT include Uri query since it has the SAS credentials; This will return the URL without the querystring. 
            // UrlDecode to convert %2F into "/" since Azure Storage returns it encoded. This prevents the folder from being included in the filename. 
            Url = WebUtility.UrlDecode(blobClient.Uri.GetLeftPart(UriPartial.Path)) 
          }; 
          docList.Add(doc); 
        } 
        catch (Exception e) 
        { 
          Console.Write(e.Message); 
          // could be specific azure error types to look for here 
          return new ResultModel<List<DocumentModel>>(null, "Could not upload file: " + e.Message); 
        } 
      } 
      else if (HasFormDataContentDisposition(contentDisposition)) 
      { 
        // if for some reason other form data is sent it would get processed here 
        var key = HeaderUtilities.RemoveQuotes(contentDisposition.Name); 
        var encoding = GetEncoding(section); 
        using (var streamReader = new StreamReader(section.Body, encoding, detectEncodingFromByteOrderMarks: true, bufferSize: 1024, leaveOpen: true)) 
        { 
          var value = await streamReader.ReadToEndAsync(); 
          if (String.Equals(value, "undefined", StringComparison.OrdinalIgnoreCase)) 
          { 
            value = String.Empty; 
          } 
          formAccumulator.Append(key.Value, value); 
          if (formAccumulator.ValueCount > defaultFormOptions.ValueCountLimit) 
          { 
            return new ResultModel<List<DocumentModel>>(null, $"Form key count limit {defaultFormOptions.ValueCountLimit} exceeded."); 
          } 
        } 
      } 
    } 
    // Begin reading the next 'Section' inside the 'Body' of the Request. 
    section = await reader.ReadNextSectionAsync(); 
  } 
  return new ResultModel<List<DocumentModel>>(docList); 
} 

I hope you find this useful as you tackle file upload operations of your own! If you have questions, or there’s another way Trailhead Technology Partners can support your business, please feel free to get in touch. And if you’d like to learn more about us, you can start your journey here

Check Out Our Newsletter!​

This field is for validation purposes and should be left unchanged.

Related Blog Posts

We hope you’ve found this to be helpful and are walking away with some new, useful insights. If you want to learn more, here are a couple of related articles that others also usually find to be interesting:

Our Gear Is Packed and We're Excited to Explore With You

Ready to come with us? 

Together, we can map your company’s software journey and start down the right trails. If you’re set to take the first step, simply fill out our contact form. We’ll be in touch quickly – and you’ll have a partner who is ready to help your company take the next step on its software journey. 

We can’t wait to hear from you! 

Main Contact

This field is for validation purposes and should be left unchanged.

Together, we can map your company’s tech journey and start down the trails. If you’re set to take the first step, simply fill out the form below. We’ll be in touch – and you’ll have a partner who cares about you and your company. 

We can’t wait to hear from you! 

Montage Portal

Montage Furniture Services provides furniture protection plans and claims processing services to a wide selection of furniture retailers and consumers.

Project Background

Montage was looking to build a new web portal for both Retailers and Consumers, which would integrate with Dynamics CRM and other legacy systems. The portal needed to be multi tenant and support branding and configuration for different Retailers. Trailhead architected the new Montage Platform, including the Portal and all of it’s back end integrations, did the UI/UX and then delivered the new system, along with enhancements to DevOps and processes.

Logistics

We’ve logged countless miles exploring the tech world. In doing so, we gained the experience that enables us to deliver your unique software and systems architecture needs. Our team of seasoned tech vets can provide you with:

Custom App and Software Development

We collaborate with you throughout the entire process because your customized tech should fit your needs, not just those of other clients.

Cloud and Mobile Applications

The modern world demands versatile technology, and this is exactly what your mobile and cloud-based apps will give you.

User Experience and Interface (UX/UI) Design

We want your end users to have optimal experiences with tech that is highly intuitive and responsive.

DevOps

This combination of Agile software development and IT operations provides you with high-quality software at reduced cost, time, and risk.

Trailhead stepped into a challenging project – building our new web architecture and redeveloping our portals at the same time the business was migrating from a legacy system to our new CRM solution. They were able to not only significantly improve our web development architecture but our development and deployment processes as well as the functionality and performance of our portals. The feedback from customers has been overwhelmingly positive. Trailhead has proven themselves to be a valuable partner.

– BOB DOERKSEN, Vice President of Technology Services
at Montage Furniture Services

Technologies Used

When you hit the trails, it is essential to bring appropriate gear. The same holds true for your digital technology needs. That’s why Trailhead builds custom solutions on trusted platforms like .NET, Angular, React, and Xamarin.

Expertise

We partner with businesses who need intuitive custom software, responsive mobile applications, and advanced cloud technologies. And our extensive experience in the tech field allows us to help you map out the right path for all your digital technology needs.

  • Project Management
  • Architecture
  • Web App Development
  • Cloud Development
  • DevOps
  • Process Improvements
  • Legacy System Integration
  • UI Design
  • Manual QA
  • Back end/API/Database development

We partner with businesses who need intuitive custom software, responsive mobile applications, and advanced cloud technologies. And our extensive experience in the tech field allows us to help you map out the right path for all your digital technology needs.

Our Gear Is Packed and We're Excited to Explore with You

Ready to come with us? 

Together, we can map your company’s tech journey and start down the trails. If you’re set to take the first step, simply fill out the contact form. We’ll be in touch – and you’ll have a partner who cares about you and your company. 

We can’t wait to hear from you! 

Thank you for reaching out.

You’ll be getting an email from our team shortly. If you need immediate assistance, please call (616) 371-1037.