Engineering at MindLink

Automate your memory profiling now

April 17, 2019

MindLink’s core product is designed to be a long running and stable server brokering between clients and chat systems. Stability is paramount to ensure users have an optimal experience. We have a large number of automated tests, exploratory and planned manual QA processes and we dogfood our products with nightly builds.

Combined, these strategies have ensured that our products continue to be scalable, reliable and high quality. But we can do better!

Recently we discovered a memory leak late into the QA and release process and this made it out into customer hands. Not ideal, but these things happen. The leak itself has implications when lots of users log on and log out over time and will eventually cause the process to run out of memory.

TL;DR

By leveraging a memory profiler that provides programmatic access to snapshots you can automate memory performance testing to give you high confidence that you haven’t introduced any memory leaks into your code under reproducible scenarios.

One such profiler for .Net code that’s free (as in beer) is dotMemory Unit.

The story

The cause of the leak could only effectively be discovered through memory profiling and it was the result of several issues combining to cause the leak, if any one of those issues were alleviated the leak would not have occurred.

If memory profiling is what it takes to notice these, or at least some manual observation, then there is always the chance that nobody actually checks (as what happened here).

Can we do better than relying on somebody to remember to check, for every release? And to check all necessary scenarios manually? Not really…

The search for a tool

What we really want is a way to automate that profiling during the release pipeline and to flag memory leaks automatically.

We could crudely invoke a process snapshot before and after an operation and then compare the memory, but that isn’t going to be reliable and how do we ensure that each scenario is actually running?

Ideally we want programmatic access to a profiler so that we can:

  1. start profiling
  2. Execute the scenario
  3. Stop profiling
  4. Make assertions on the profiling session

As luck would have it, such magic exists! We leverage the .net framework and we are therefore fortunate to be able to benefit from the brilliant dotMemory Unit project by JetBrains (the makers of ReSharper and Kotlin).

Unlike its GUI profiling counterpart dotMemory, dotMemory Unit is free! And it lets you programmatically profile and make assertions on the profiling session.

There are some caveats:

  1. Tests have to be synchronous
  2. The test runner has to be called via the dotMemory Unit executable.

That seems reasonable considering the power that it gives us. With dotMemory Unit we can write automated and isolated tests that ensure we don’t have memory leaks.

Writing a memory unit test

For our issue that snuck under the manual testing radar we knew the specific scenario that showed the issue, all we needed to do was log a user onto a chat system and then log them off again, basically the simplest scenario we could write - so that’s nice!

Here’s an example NUnit test:

[Test]
public void LoggingOffCleansUpSession() {
  this.LoggingOffCleansUpSessionAsync().Wait();

  dotMemory.Check(memory => Assert.That(memory.GetObjectsOfType<ISession>().ObjectCount, Is.EqualTo(0));
}

private async Task LoggingOffCleansUpSessionAsync() {
  var session = CreateSession();

  await session.LogOnAsync();

  await session.LogOffAsync();
}

What we are doing here is creating a session, logging it on and then logging it off. Since the session is asynchronous and we want a clean test case we use an async method to perform the test steps and wait for that in the actual test to satisfy dotMemory unit’s synchronous constraint.

Our first use of dotMemory Unit was to create a test over the entire session stack, since the memory leak we encountered was the combination of internal behaviour and how the session was wired together. This high level test will also flag any memory leaks specific to individual units.

We are aiming to create memory tests for more isolated areas of our code so that we can have better coverage over different scenarios. It’s a work-in-progress, but certainly an improvement on relying on manual testing and something I think everybody should consider when stability and performance is important.

In a future post I will discuss how we integrated dotMemory Unit tests into our Azure DevOps release pipeline.


Luke Terry

Written by Luke Terry.

Senior Engineer at MindLink. Enjoys technology, playing games and making things work, blogs at www.indescrible.co.uk.