Tag Archives: Redis

Creating High Performance Redis based Analytics for .NET Applications

Recently working on exception and logging in a Ruby project I came across a nice gem called Minuteman which uses Redis bitmap to track user activity, further investigation shows that Redis can be used to track other metrics of your application very efficiently as well, last weekend I started to port in C# and thus Minuteman .NET version was born and nuget package is also available. Behind the scene it uses StackExchange’s BookSleeve as a Redis Client and as an added feature I have also utilized Redis built-in pubsub so that Signalr can be used to report real time update.

Minuteman contains UserActivity and EventActivity classes to track and report user and application events. Lets start with the UserActivity and see how it can be used.

Lets say, you want to track when a user logs into the system, to track:


              await userActivity.Track("login", CurrentUser.Id);
              

Later on, you can check whether the user has logged into the system in a specific time frame like:


              await userActivity.Report("login", ActivityDrilldown.Hour).Includes(CurrentUser.Id);
              await userActivity.Report("login", ActivityDrilldown.Day).Includes(CurrentUser.Id);
              await userActivity.Report("login", ActivityDrilldown.Month).Includes(CurrentUser.Id);
              

The time frame is supported from second to year, by default it is Hour, multiple User Id can be also passed in the last argument.

You can also use it to check how many users have logged in the last hour like:


              await userActivity.Report("login", ActivityDrilldown.Hour, DateTime.UtcNow.AddHours(-1)).Count();
              

There are other interesting bit operations that you can do, for example to get the users who bought both apple and banana:


              var apple = userActivity.Report("bought-apple", timestamp);
              var banana = userActivity.Report("bought-banana", timestamp);
              var count = await (apple & banana).Count();
              

Or either of it:


              var count = await (apple | banana).Count();
              

Or either of it but not both:


              var count = await (apple ^ banana).Count();
              

Behind the scene it uses Redis GETBIT, SETBIT and BITOP, so the result is extremely fast and since it uses bits to track the activity the data consumption is also very low.

UserActivity is best to track booleans but there are cases when we need to track how many times a certain event has occurred, in those cases the EventActivity comes into action. Lets say in your application you may want limit your api usage 60 calls per minute:


              var key = "api:usage:" + clientApiKey;
              var now = DateTime.UtcNow;
              var count = (await eventActivity.Counts(
                  key,
                  now.AddHours(-1),
                  now))
                  .First();
              
              if (count < 60)
              {
                  // Your code
              }
              
              await eventActivity.Track(key);
              

Both UserActivity and EventActivity has EventNames method that returns all the event names that you used for tracking. There is also a Reset method which you can use to erase the existing data.

To use the Pubsub both the class has a CreateSubscrption method which accepts the same event name that you pass in the Track method, it returns a Subscription objects which you use to Subscribe and later on Unsubscribe. But for the subscriber to comes into action the publishable argument of the Track needs to be true.


              var subscription = eventActivity.CreateSubscription(
                  "my-event",
                  e => Console.WriteLine(
                      "{0} {1} {2}", e.Timestamp, e.EventName, e.Count));
              
              
              await subscription.Subscribe();
              
              // In another part of your app
              await publisher.Track("my-event", publishable: true);
              

I have also included a Test Application that shows how Signalr can be used to show a histogram of live data. The following is a screenshot of it:

Minuteman realtime update

That it for today, your feedback is always welcome.

Refrence:-

http://kazimanzurrashid.com/posts/creating-high-performance-redis-based-analytics-for-net-applications

Advertisements