In the last post I wrote about ways to tackle bottlenecks and to increase the performance of an application. When dealing with a high traffic application servicing tons of concurrent users all requesting the same data, intelligent caching can boost the speed in which requests are processed. As a result, lots of load is taken away from the database.

Querying a database makes much sense when the data requested changes a lot. But imagine the database is backing a blog or news website where the data requested from the database is the same in hundreds or even thousands of cases. The most frequent query sent to the database in such an environment is probably for loading the article on page one, which - once published - probably won’t change a lot. This doesn’t hurt the Database if your private blog has 100 unique visitors per day, but if the frontpage of a popular news website is attacked by 100.000 unique visitors per hour then you are in trouble. You need to use caching in order to take away stress from your database server.

An excellent, widely used open source cache solution which is easy to use and yet powerful is OSCache (http://www.opensymphony.com/oscache/). Besides features like clustering, a servlet filter for caching entire JSP pages or persistent caching it comes with a JSP tag library that enables you to include caching into your jsp based web application so easily it hurts.

All you have to do is:
1)       Add the OSCache jar file to your classpath, put the tld file into the WEB-INF directory of the webapp and declare the OSCache taglib in your jsp page:

2)       mark portions of the jsp-page to be cached in your jsp page with cache tags:

In this example the cached portion of the page is stored under the key defined in the variable cacheKey, it supports multiple languages and expires after 1800 seconds. There are a few other attributes available for configuring the cache tag. More information can be found here http://www.opensymphony.com/oscache/wiki/JSP%20Tags.html

A cache with an invalidation timeout as seen in the above example is great for many situations, but sometimes you need full control on when a cache entry needs invalidating. OSCache offers an API to manage the cache in your Java code. This is an example of how to invalidate it in a Java web application:

Piece of cake!

Take care,

Alex

PS: OSCache is under the OpenSymphony Software License, Version 1.1 which is derived and fully compatible with the Apache Software License.

The performance of a dynamic web application can be quite unpredictable when you don’t know how many People are going to use it and of course how they are going to use it. You can predict some of the users’ behaviours and do your best to prepare your application to cope with the expected load. But sometime after the launch you suddenly find yourself in a situation where the performance of your application – or at least parts of it - is poor and the customer is unhappy. In cases like this typical reasons why you didn’t find the bottlenecks while developing are:

• more users than expected
• unpredicted behaviour of users
• the amount of data stored in the production database compared to the development database
• missing indexes on database tables
• slow queries that need optimizing besides adding indexes
• inefficient code
• insufficient resources on the servers

So before using the brute force approach by throwing more hardware at the problem, the most important task to accomplish first is to find the real bottleneck. Just like in war times good intelligence is your key to winning the battle. So gather information first.

• Find out which parts are slow and under which circumstances the performance is poor. Talking to actual users of the app is a good approach to do so.
• Monitor the servers using tools like munin. CPU and Memory usage can show you which part of the architecture is stressed. In some cases the webserver idles most of the time, while the database machine is under heavy load
• Check the Database for slow queries – most databases provide tools and statistics to determine the slowest queries. This will help you find out where indexes might be missing or which queries need optimizing
• When optimizing queries first look at the explain plan. It can point you to the problem directly.
• Check the DB for the most frequently executed queries – if a query is executed very often this might be a place where the application code needs optimizing or where caching might help. Add caching wherever the same piece of infrequently changing data is shown to many users many times. A great roundup on open source Java cache implementations can be found here.
• Use a profiler to find out where in your code the application gets stuck. Chances are high that you end up locating the place where one of the slow queries is executed

Then start optimizing the parts you expect the most performance boost from. Be careful though when adding indexes on database tables with huge amounts of data – check if there is sufficient disk/table space available on the RDBMS. Indexes can also have effects on the performance of write operations.

If all this brain doesn’t solve the problem you might need to add muscle to your hardware:

• Scale vertically by adding RAM and increasing CPU
• Scale horizontally by adding machines, which is a lot easier for web servers as it is for database servers
• An excellent piece of literature on this topic is “Scalable Internet Architectures” by Theo Schlossnagle

Take care…

Alex

Let’s face it: we hate writing proposals. We’re engineers and we want to do what we do best: write code, design great architectures, try out new technologies, stuff like that. Nevertheless proposals are extremely important: no proposal -> no project -> no work -> no pay -> no doughnuts.

So here are my rules for writing a good proposal:

1) Embrace Proposals
Especially in times like these, where every customer seems to be cutting back costs, we should be happy about any request for a proposal that comes in. And with this positive attitude our proposals will get a lot better. The customer is our friend. We need her more desperately than the she needs us.

2) Think win-win.
Don’t be too cheap, don’t be too expensive. Just be fair. Customers are smart – they know if they can trust us or not. Both being too cheap and being too expensive may seem like loose-win or win-loose at first sight, but will lead to a loose-loose situation in the long run. If you’re too cheap, the customers get used to your prices and you won’t enjoy working for them anymore. Being too expensive will lead to unhappy customers, because they won’t trust you and find someone else for the job. We are exchangeable.

3) Don’t just think in terms of man-days.
Simply thinking in terms of man-days is not too bad. If estimated properly the project will cover your costs and maybe leave you some profit on top. But imagine you can finish a complex project in a short amount of time, because you already have this great component you’ve written in a previous project that you can use for this one, too? Ask yourself: how much is it worth? How much would others charge for the same project?

4) Cover your back
Always have someone else check your proposal. Never send a proposal to a customer, no one else has taken a good look at. Ask someone to check the proposal critically, question everything from the concept down to the pricing. The resulting proposal will contain less typos and someone else might find something significant you might have forgotten or overlooked. Maybe someone else will think it’s too cheap, too expensive, lacking detail or whatever.

5) Be complete and be clear
The clearer and the more complete the definition of the deliverables, the less potential for discussions with the customer there’ll be. Not only mention what you will deliver, also mention what you will not deliver. If the proposal is about writing a custom intranet application, be sure to mention that setting up the production and staging environment is not part of the proposal – if, of course, someone else is in charge of that. Sometimes, out of laziness, we write stuff like “standard reporting – 1000$”. The customer will probably have a completely different opinion on what a standard report is than you. Sentences like this have the potential of eating up your profit, because the customer expects you to make him happy. And instead of a CSV export of customer addresses, you find yourself writing a full featured CRM application.

7) Find the pitfalls
Every project contains risks. Think about worst case scenarios and everything that could go wrong during the project – e.g. introducing a new technology you haven’t worked with before. The more risks you find, the more buffer you add to the price. Sometimes it even makes sense to mention the risks and the customer will better understand the pricing.

7) Blurry details, blurry price
If it’s not completely clear what the customer wants and what the desired outcome of the project will be, you’re walking on thin ice with a proposal. In that case it’s often better to name ballpark numbers and ranges instead of a fixed price.

Is there anything I’ve forgotten? Let me know and leave a comment – I’ll appreciate it.

Alex

One part of reviewing and refactoring code is finding and removing code smells. An example, which I encounter quite often is the outmoded manner of coding Enumeration-like behaviour using int-constants. Or worse: String constants. The java.util.Calendar-class serves as a great example for this former best practice, before the Enumeration-Type was introduced with Java release 1.5.

Imagine you want to get a java.util.Date object pointing to the 1st of January 2009, midnight. This is one way of doing it:

This has been a clean way of implementing and “imitating” enum-like behaviour in pre 1.5 times and of course in projects that require a lower java version.

Since we’re passing ints to the set method, it is possible to do something like this, without having the compiler complain about it (seen it done before):

And due to the fact that the number of the months in the Calendar class starts with 0 this would create a Date object pointing to the 1st of February. Ouch.  An even more evil way of misusing the Calendar class could be this:

I’ve never seen this done before, but looking at it, I must say it really hurts.

Since the release of Java 1.5 there is a much better way of programming that kind of behaviour using the Enumeration type. This is how a simple Month enumeration could look like:

But we can add a little more brain to the implementation:

We’ll add member variables for the name, index and number of days, a constructor with arguments for each of these values and we’ll provide getters for retrieving them. Additionally we’ll add a method for calculating the correct number of days depending on a given year, considering leap years.

And here’s an example of how the enumeration can be used:

I warmly recommend the use of this great feature of the Java language. It definitely adds more type safety, ease of use and security to your application. But unfortunately it seems to me, that it still hasn’t reached the popularity that it really deserves, yet.

Whether you’re part of an agile team using methods like XP or Scrum, or processing your projects in a more classic way with RUP or even waterfall, in most cases you’re part of a team of developers. Chances that someday you’ll have to work with or (worse) debug code someone else has written are 100%. And who of us hasn’t complained about having to deal with “the mess” someone else has made. And probably some of our co-workers have been scratching their heads with both hands looking at some piece of code we’ve written after midnight with a numb mind in shipping hell. Nobody’s perfect.

So what can we do about this situation? Whining, complaining and pointing fingers won’t make the code any better.

Just recently I read “Clean Code” by Robert C. Martin. Zillions of blog entries were recommending this book, so I had to get it myself. I was probably one the last people on earth who hadn’t read it. It’s a great piece of programming literature. Reading it I found myself nodding my head a lot, agreeing with many of the points made and regretted I hadn’t read it earlier. There are some aspects I do not agree with 100% but they were very few.

One thing that impressed me a lot because of its simpleness is the “boy scout rule”: “Leave the campground cleaner than you found it”. When working with a piece of code, make your changes and commit it a little cleaner to your version control system. Let me give you an example.

I was debugging a class for creating financial charts. The following piece of code was scattered in it with slight variations about 20 times.

The reason I was debugging the class was that the font sizes within the charts did not look right when run on the production environment. Simple change made: remove the hard coded String and the numeric font size, and replace them with constants.

No big deal, but improving the readability and maintainability of the code a little bit. Imagine a whole team following this simple principle. More productivity, less rotten code, better team spirit. And the best thing about it: it’s absolutely free!

Happy coding!
Alex

I spend about one hour a day commuting between my office and my home by car. During drive-time I used to listen to the radio a lot. Someday the flooding of air time with commercials drove me nuts, so I decided to change something about that. After having spent some time with podcasts I found out that audiobooks are the better alternative for me: they are professionally produced,  you can read the reviews on them on amazon before you buy them and they usually - when selected wisely - contain less crap than some podcasts (I am a podcaster myself (www.de-lite.de), and I’m sure many people consider my productions crap, too). The downside is - audiobooks are not free

Anyway - just recently I downloaded and listened to “Getting Things Done: The Art of Stress-Free Productivity ” by David Allen, a book everybody I know obviously had already read, except for me. When I first heard of GTD a couple of years ago, what first came to my mind was some kind of car, a Gran Turismo with a Diesel engine or something like that. Then I learned it was a methodology for self management. Some people I know claim to be using GTD as a method for self-management. Now, looking at some of their desktops (both digital and analog) I know they aren’t.

Many poeple misunderstand GTD simply as using a specific digital tool to store and manage all their todos. In fact it is more of a methodology or even a philosophy on how to organize both your professional and private life. And the toolset for GTD can vary from person to person. In order to be doing GTD of course you need tools, but those can be completely analogue. A printed calendar, a stack of paper, a pen and a bunch of file folders will suffice for some people if used properly.

So what is this GTD thing all about? I’ll try to outline the basic steps of the methodology in my own humble words (and probably missing a lot of points - sorry in advance!):

1) Collect

• collect your thoughts, what’s bothering you at the moment, get it all out of your brain and transfer it to a physical system (e.g. a sheet of paper). When doing this for the first time, it will probably take you hours to finish that process. The good news is that once you’ve got all of your projects and tasks together, adding the additional stuff that pops up each day won’t cost you a lot of time
• stuff like email does not need to be collected. It’s already stored in your inbox. It just has to be processed - and that’s easier than you might think. Before I started following the rules hundreds of emails were lurking around in my inbox.
• the aim is to free your “psychic RAM” from all that’s in there. This will in turn free a lot of energy and take away stress from you. “It’s a lot easier to relax when you know what you’re NOT doing at the moment” is a proverb David Allen uses a lot. And as I experienced this is so true.

2) Process

• look at everything that you’ve collected and decide what you have to do about it
• some of the items are actionable and won’t take longer than two minutes, e.g. “answering that email” or suchlike. Do those immediately and get rid of them
• in case the item is not actionable, consider if it might be actionable someday maybe -> put those on a separate list
• in case the item is not actionable, consider if it might be worth keeping -> throw it away if not, otherwise file it

3) Organize

• assign the actionable items to projects, groupe them by context (Telephone, Office, Home, En route), project, time and energy needed
• if an action needs to be processed on a specific point of time or day in the future, add a reminder to your calendar
• by the way: projects are things where more than one action is needed to accomplish them.

4) Review

• Review your lists every day
• look into your calendar first thing in the morning, so that you know what definitely has to done that day
• Every week reserve about two hours time for reviewing and rethinking your projects, lists, tasks and thoughts from different angles (also the big picture)

5) Do

from the list of actions chose the best one to be done next. This can vary depending on:

• the context: are you online? have you got access to a phone? are you in your car for making errands? etc.
• time available: if you’ve got a ten minutes time slot before a meeting, choose a task you can finish within that amount of time
• energy: if you’re tired, choose a no-brainer, otherwise something more sophisticated.

The audiobook cost me 30 bucks and it was one of my best investments ever. I’ve been following the rules for a week now and my impression is, it seems to be working fine for me. For the first time in years the number of emails in my inbox is 0, NULL, zip, nada. Pure joy! It even made me start this blog - something I’ve been thinking about for months but never got started with.

In one of my next posts I’ll write about my GTD toolset and how I work with it every day.

Take care,
Alex

David Allens website
The book on amazon
The audiobook
about GTD on wikipedia

PS: feel free to leave a comment, if you liked this post. If not, please leave a comment, too and let me know where I can improve. I appreciate it!

Anyone who ever seriously started writing unit tests will sooner or later have come to the conclusion, that it helps isolating small testable chunks of code resulting in a clean structure of methods within a class and forcing the developer to write cleaner code. This is easy as long as the tested code does not work with data or configurations that might change depending on the environment in which the test are run.

My two favourite basic rules for writing tests (Thanks uncle bob!) are

• tests must run anywhere. (e.g. on a continuous integration server)
• tests must run fast, so they’re fun running them.

As soon as your tests depend on a database containing specific data or a file system with a certain directory structure you’re in trouble. In my experience many programmers write test for methods that are easy to write tests for, and avoid the more complicated cases requiring setting up test data and such like.

A really cool way for decoupling tests from dependencies like this, speeding up their execution time and finally leading to a much better code coverage are mock objects.

A mock library I really enjoy using is easymock. Let me show you an example of how easy easymock (http://www.easymock.org) really is.

First include this library to your classpath: easymock.jar

In case you want to mock classes that do not implement an interface, additionally add this library: easymockclassextension.jar

Now you can start writing your isolated test making use of mock objects

Add this include to your test class:

or this if you’re mocking interfaces:

This is how a test could look like (example code)

Mock Objects will definitely help you decoupling your tests from platform dependencies. They will make your tests faster and more fun to run.

Alex