Web developer from Sydney Australia. Currently using asp.net, mvc where possible.

Tuesday, June 22, 2010

Introduction to StickyBeak

While working on Jobping we wanted a raw record of each request made to our site so IF something happens to go wrong we would have all the data necessary to recreate the event and/or the data itself. Needless to say that it has proved very useful to investigate what has happened on the site.

However the code used for this logging is embed into the main project and not easily portable, I wanted to make a assembly that captured this functionality so I could easily drop it into the next project. So we created  StickyBeak.

StickyBeak is a logging tool for asp.net websites written in c# and currently requires the NLog logging library to run. StickyBeak’s purpose is to log each request to your web server and also provide a easy interface to view these requests.

Looking at these logged requests is extremely useful when you are trying to find the cause of an exception or  even more useful when you are trying to recover some lost data because of an exception.

StickyBeak works as an HttpModule and logs the raw request data into a log file using Nlog. The information recorded for the requests includes, date, http method, url, User.Identity.Name, IP Address, unique session Id, unique browser Id, header values, querystring values, posted form values and cookie values.

Below is a screenshot of the admin viewing tool, which lets you see the logged activity on your site.


How it works

StickyBeak runs as a HttpModule, each time a request is processed by .net the module creates a new RequestLog object and populates all the data using the current request. The RequestLog object is then passed to the LogRepository which saves the Requestlog object.

Currently there is only one LogRepository, this repository uses NLog. The NLogRepository writes the LogRequest object out to the log files in a custom format. The NLogRepository can also read LogRequest objects for viewing using the admin interface.

Configuration Needed For StickyBeak To Work

  1. You need a reference to the StickyBeak and NLog assemblies contained in the binary zip file distribution on CodePlex
  2. Configure the StickyBeak HttpModule

  3. Configure the handler (to display admin interface). The configuration below also secures the handler you may wish to remove this for testing.


  4. Configure NLog to record the logging information that is record by StickBeak


  5. You can also optionally configure exclusions for StickyBeak. For example you could exclude all requests to a certain URL, exclude a querystring/form/cookie/header value by key etc. See the sample configuration for more details.
You can download the source and binaries from StickyBeak on CodePlex. kick it on DotNetKicks.com Shout it

29 comments:

Jerome said...

This looks really useful. Any thoughts about logging Session variables too (possibly with the ability to choose which ones to include / exclude)? :)

Mark Kemper said...

@Jerome, thanks for the feedback. I just plain forgot about Session because we don't use it. I have added this to the codeplex Issue tracker so we can tackle this in the next release.

ilivewithian said...

Can you do a compare and contrast to ELMAH, your tool looks interesting, but I'm not sure I see a compelling reason to move from one to the other. Or am I missing the point and they are two complimentary tools?

Mark Kemper said...

@ilivewithian Yes they are complimentary.

ELMAH logs exceptions only, it logs stacktrace + exception messages, just as your would see on the YSOD.

StickyBeak on the other hand logs information such as url, querystring, posted form values, cookies etc for EVERY request.

For example: when an exception is logged with ELMAH you can't actually tell what the person was doing just before (you only have some limited details on the request that caused the exception).

With StickyBeak, you can find that particular person's session and see exactly what they were doing at the time of the exception and leading up to the exception.

StickyBeak also logs much more detail about the request and doesn't concern itself with exceptions at all.

Hope this helps.

Mark

Roni said...

the project code looks very promising - do you have also wrote a player for your recorded data? this would be very useful if you like to make performance test with real scenarios.

Mark Kemper said...

@roni thanks, however I am not sure I understand you comment completely.

What do you mean by player? You can view the recorded data using the web UI or do you mean something to 're-play' a request against the server?

Performance tests would be good to have in the project. Although performance results / trade offs would need to be assessed for each individual scenario where StickyBeak is used.

mreyeros said...

Definitely looks like a promising project/tool. What are the implementation steps for an ASP.NET MVC project?

Mark Kemper said...

@mreyeros the config is exactly the same for a mvc project. It runs as a httpmodule + a httphandler.

Chris Fulstow said...

This is great. Next step could be to build another HttpModule that analyses the data captured by StickyBeak and detects then blocks requests from unwanted web crawlers.

Richard Dingwall said...

Very cool -- does it scale with a high volume of requests though?

Sam said...

Any plans to add support for logging to a database??

I would love to be able to use this on an existing web farm, but I don't really want to have to pull the different log files together manually.

Mark Kemper said...

@Chris Fulstow yeah there are heaps of possibilities for using the data collected. Blocking spam bots and such would be a whole new project but it would be good to analyse this data offline and then suggest/add new rules to the blocking system.

Mark Kemper said...

@Richard Dingwall it will scale out as it currently logs to the file system, however there will be overhead and the performance hit will need to monitored on a individual basis. There is also a management issue with all the log files that would be generated on a large scale system.

Mark Kemper said...

@Sam yes. This would be my next feature. Something to manage the log files generated. Currently, the code is setup to easily switch out the file system loggers with something else such as db.

However I am thinking more down the lines of a console app that would process the log files and move then into the database. Just to keep it as learn as possible.

Having a database repository would make both these tasks trivial.

Chad said...

Thanks for putting this out there. Very promising, especially during integration and beta testing.

chiragnirmal said...

Hi Mark, StickyBeak looks like a very useful utility. We also currently have a logging module which we built inhouse for logging and emailing debug information, but it does not track as many variable as Stickybeak.

We also use a 3rd party cache and session repository called ScaleOut server. ASP.net automatically stores all session variables into ScaleOut based on web.config setting, but it does not do the same for cache. and I was wondering if there a way to configure 3rd party cache repository into StickyBeak?

danfindley said...

Does this work on IIS6? I guess I could download it and try it, but it's easier to ask :-)

Mark Kemper said...

@chiragnirmal there is no built in support for third party caching. It current only logs requests to the filesystem ..

Mark Kemper said...

@danfindley of course, its targeted at 3.5 framework. Just make sure the handler url ends in .ashx(configure this in web.config) so iis 6 will route request to .net. But I have not tried it!

Also it will only log requests that go through to .net (unless you set iis up specifically)

Amit said...

Great feature. Thanks.
Couple of questions/suggestions:

a. What is the performance impact? say, if the session/page is bulky. Can the logging step be made async to the request processing?

b. Is the information logged configurable? May be it can be read from a schema file and can be configured from the admin tool?

Dotnet said...
This comment has been removed by a blog administrator.
Fatal said...

hi, this looks like a pretty cool tool.

I have a couple of suggestions:

1) It would be handly to be able to enable/disable it via the web interface - obviously it would revert to the web.config settings once the app restarts but I can think of cases where you might want to switch it on for a while to try to track down an issue and you dont want to restart the app.

2) It would be useful to make the viewstate a hyperlink that goes to another page which displays a nicely decoded and formatted representation of the viewstate.

Mark Kemper said...

@fatel thanks that's two great suggestions. Will add them to codeplex.

On holidays at the moment bit will look to release another version in a about 3 weeks and add some more documentation.

seoinheritx said...

Your posts are so cool. Please continue this good work.

Ruby on Rails Development | Asp.net Development

Sumo Digital Incorporation said...

this blog is really interesting and gives good details.

Weighing Scale | Weighing Scale Supplier

Dhiraj Siddhapura said...

Your blog is very appreciate and helpful.

Mechanical Seals

Asava Samuel said...

This is even better than NLog:
http://www.kellermansoftware.com/p-14-net-logging-library.aspx

Govind Sharma said...

Its having good description regarding this topic.It is informative and helpful.I have known many information from this.

Thanks for sharing.

Mechanical Seals

Sudhir DBAKings said...

Nice post very helpful

dbakings