If you’ve heard me speaking at any of the worldwide conferences, you’ll almost certainly remember at least two things about me. 1: I’m British. 2: I’m somewhat obsessive about web log analysis. And I always like to squeeze in at least one extra fact. Or two.
In this day and age, no company has any good reason for not regularly analysing their server logs. And don’t make the mistake of thinking that your server’s free online stats are enough. They’re not.
The bottom line is that without log analysis you are blind to any threats, opportunities, weaknesses and strengths of your website. A reverse SWOT of sorts!
Imagine that your business wasn’t online, but existed only as a physical concrete presence on a high street. Can you imagine not knowing how many people come into your shop each day? Not having a clue what they look at, where they go, and how long they spend before leaving?
Supermarkets realised the importance of this information years ago. They know how to setup their stores to make sure that their consumers buy every single item they want, and then a whole lot more besides.
With detailed analysis of your web server log files, you can do the same with your website.
On a typical working day, I probably look through between five and ten sets of server logs. I won’t tell you how many times I do this at the weekend. You won’t know whether to pity me or be impressed.
But by the time you get to the end of this article, I hope to have shared some of the knowledge that I’ve acquired, in the hope that you too will realise the pure gold that lies within your log files, and how to tap into it.
Before we dive in, a few important pieces of information. Let’s start with two reality checks.
Reality Check 1: If you were to run the same set of log data through five different web log analysis applications, you would get five quite different sets of reported results.
I’ve tried more web log analysis applications that you probably realise exist, and the range of different figures for the same “facts” can be staggering.
The good news is that the better ones more or less agree with each other. Which applications are the accurate ones? Bear with me. We’ll get there.
Reality Check 2: Log analysis is time consuming. No matter which application you use, it takes time. Which is why so many small (and not so small) companies put it off. Bad idea.
Another potential stumbling block is the terminology. So let’s be clear here:
A hit is a request for a file from a web server. If my main index page consists of one HTML page with five images, one person coming to view that page will generate six hits. 1 HTML + 5 images.
Visitors – you can probably work out for yourself. But some log analysis software tries to recognise unique/individual visitors. For example, my RSS reader may hit your feed ten times a day. Some log apps will count that as ten visitors, some will count it as one.
IP Addresses are also important for most log analysis applications, but may be misleading. Think of people sharing IP addresses on a network, dialup accounts, non/fixed IP addresses and so on.
Aside from terminology, there are also three other important points to consider.
Point 1: If a website (for example a software site) pulls in images or data from your own website, this may show up in your logs as a visit. So if Dave’s Cool Site features your app on their front page, and displays a screenshot that is being pulled from your server, Dave’s Cool Site’s 10,000 visitors may look like your visitors in your logs, even though they never visited your site.
Point 2: DOS attacks, spam and hacking attempts sometimes produce all sorts of strange entries in your web logs.
Point 3: Download managers can make a real mess of your download counts or hits to your file. If a download manager creates ten simultaneous connections to your EXE file, this will probably show up as ten hits to the file in your logs. The solution? Divide the bandwidth to each file by the size of the file. That will give you a more accurate figure.
One more thing. I warned you I try to squeeze more information in. Always remember that the period of time you are looking at is important. As a general rule, don’t make decisions on information gleaned from anything less than a full week or two of data, and it’s often a good idea to work in units of seven days. It’s also important to take into account any regional, national or international holidays, festivals, religious events, major sporting events and seasonal trends. Always take a step back from your data.
Enough of the provisos. Let’s get into what I look for in our client’s logs.
Every log analysis app has some sort of overview or summary section. This is a useful snapshot of what’s going on, but in general isn’t worth spending too much time on. The total number of visitors isn’t as much use as how many visitors came on each day, the average page views is utterly meaningless, and the time spent? Don’t even bother reading it.
So what do I look for?
Referrals are obviously of great interest. You want to know who is sending you traffic, whether there are unexpected or unknown names in there, and of course whether there have been any changes and/or trends emerging. Falls or drops in referrals are just as important as rises.
Sites sending two or three visitors a month aren’t worth a second glance, but any generating a significant number of visitors should catch your attention.
You may, for example, find that your company has been discussed on a forum. You may also notice a crack site generating high levels of traffic. The choice is yours as to how to respond to these situations. But without analysis you’d never even know.
Interestingly, we once spotted one of our competitors who had copied and pasted the text from our own site right on to their front page. How did we know? They also copied the link to one of our pages. Don’t you wish you had competitors like that?
Aside from general referrers, many log analysis apps also offer specific information related to which of the search engines are sending you traffic. It’s important not to make the mistake of thinking that only Google matters. If they have one of their renowned/feared updates and pull you from their index for no apparent reason, you’ll wish you’d covered other options. If your website is getting no traffic at all from MSN and Yahoo, then you might want to look into it.
You may also like to drill down a little, and see which of the pages on your website perform well on the engines. Why? Because if or when you make changes to your site, you want to be careful not to delete or drastically alter any of your high performing pages. Again; no analysis means you would never know.
Entrance pages are also extremely important, and surprisingly often overlooked. Most companies will find that the most popular entry page is their home page, but this isn’t always the case. If people are arriving elsewhere, it’s important to make sure that they’re seeing all the relevant information.
Equally important is seeing what they do next. Which pages they go to and which links they click on. Are they behaving as you want them to? By moving links around, playing with different wording, graphics and calls to action, you may be amazed to see how much control you actually have over your visitors. Put this to good use.
Exit pages are equally important, and just as often overlooked. The reality for many companies is that the most popular entry page will also be the most popular exit page. But not always.
And there may be other pages on your website that instead of feeding your visitors with the information they seek, effectively act as black holes, swallowing them up forever. If there are such pages, and they receive a fair number of visitors, then you need to understand why and seal up these holes.
Also bear in mind that different log analysis apps handle the issue of exits differently. Some may report a click on a download as an exit, some may count a click on a mailto link as leaving the site, and if your files are hosted by a third party, then all of them will look at these clicks as a visitor leaving.
The paths that your site visitors take are also of great importance. With a little digging and detective work, you may realise that certain sections of your website are more or less hidden, or that your visitors are having to work hard to find what they are looking for. Hint: They won’t. They’ll give up.
There are also what I call the “basic stats”: which countries your visitors come from, what operating systems they use, what screen resolutions and so on. But bear in mind that as interesting as it may appear, this is often highly inaccurate. Country information is notoriously imprecise, the browser and resolution may be on their work/school/library machines and so on.
Errors, however, are far more important (and accurate), and again, to my constant surprise, these too are often more or less ignored.
If a referrer of a significant number of visitors has the wrong URL, then believe me you’ll want to know about it. A company that we worked with sent out a press release, and got a great write-up in a newsletter. Unfortunately the editor mistyped the URL, and anyone clicking on the link only got a 404 error. How’s that for a missed opportunity?
Note also that incorrect URLs may be from external or internal links. And that in general, there will be a variety of “odd” errors, mostly generated by attempted hacks and exploits. Unless there is a massive number of them, you can probably ignore most of these too.
And if issues such as partial content and bad requests are common, you’ll want to get to the bottom of it as quickly as possible. Find out why and solve the problem.
Finally, how to make your life easier. I belong to a certain association of shareware professionals (you know the one), and every now and then someone shows up who still reads their log files with Excel. I always assume that this person has far too much time on their hands and views their logs as a curiosity, and not a valuable resource.
The fact is that there are numerous good applications out there, two of which I can personally recommend.
The first is ClickTracks at www.clicktracks.com. In a nutshell, it’s the single most powerful log analysis application that I have ever worked with. Nothing else even comes close. A few weeks ago there would have been a “but”. A big “but”; the price. But they’ve just released a free version of their software.
I’m not going to get into it now, but trust me, you want to take a look for yourself. This is a major time saver and a powerful tool.
The other app I recommend is Web Log Storming – www.datalandsoftware.com. It has a very unique approach that allows you to really dig deep into your logs in real time. I can’t recommend this software enough, and at $129 I rate it as a steal.
The bottom line is that too many companies neglect their logs on an ongoing basis. If you’re one of them, I can’t stress enough just how accurately you may be shooting yourself in the foot. You may be oblivious to mistakes, be losing sales, be throwing away prospects and leads, and be wasting money on advertising in the process.
It usually boils down to time. There isn’t enough of it. I understand. But this is just as important as anything else you do in your work hours, and probably a whole lot more important than some of the other things you do.
Make the time. At the very least schedule two hours on the first Tuesday of every month. Stop losing sales.
—
CAN PEOPLE TRUST YOUR WEBSITE?
When your website is your main place of business, credibility and trust are two vital components. Yes, your software is the main focus of your site, but you can’t expect your customers to be prepared to part with their money unless they get a good feeling about you, too. This is particularly important if you’re a one-man-show or a small business – and it’s often the little, easily overlooked things that matter.
So how do you make people trust you? We’ve put together a list of our top five credibility-musts – ignore them at your own peril.
1) Don’t be a faceless programmer. People prefer to buy from other people – buying from a bland Internet-drone is not going to give them any satisfaction. Tell them your name, show them your picture (even if it’s not great!), tell them a little something about yourself. If it comes down to choosing between Mike who loves kayaking and has a Labrador, and some unknown person whose name appears to be sales@trulycoolsoft.com , most people are going to choose Mike.
Make sure you’re easy to contact, too. Include a phone number, even if you’re not always going to answer it. A fax number is another must, and of course a real physical address. Nobody wants to give their money away to some cyberstranger – so let your customers know you’re a real person with a real location.
2) If you’re an expert, say so. This is something that many sites ignore, and it’s a real shame. They talk about their company, their mission statements and the type of software they create, and forget about the skills and knowledge of people who actually work there. If you are good at something, tell your customers. If you have a degree, if you’ve published a book, if you’ve got 20 years of developing behind you, let your visitors know – anything that shows your customers you’re experienced in your field helps to build trust.
3) Include testimonials, but only if they are real and easy to check. Anyone can write a fake testimonial and sign it “John, Kansas” – but solid, verifiable testimonials are incredibly powerful sales tools. Include full names and locations, and if at all possible links to corresponding websites. People love quotes, but they’re not stupid – they know you can write them yourself.
4) Make your website look professional and up to date. How can you expect anyone to have faith in your software if you’re unwilling to spend time or money on your place of business? If you’re not a designer, employ one, or at the very least buy a nice template. Seriously.
Similarly, do your utmost to avoid typos and spelling mistakes, because they really do make a terrible impression. If your English isn’t up to scratch, get someone to help you. Think about it: would you shop in a store with dirt in the corners, broken light bulbs and peeling paint? No, and neither would your customers.
5) Show restraint with adverts – if at all possible, avoid them completely. Having AdSense plastered all over your site makes it look both ugly and unprofessional. The worst case scenario is that your customers are lured away by the ads and forget to return to your website. The best case scenario is that they can see you’re not making enough money from your software. Neither are great options. Avoid ads.
If you’re selling your software online, you probably look at the web as a fantastic sales, marketing and publicity machine. But many of your customers are all too aware of the risks and dangers of parting with their cash through their web browser. Do everything you can to reassure them.
—
ON A LIGHTER NOTE:
In a few weeks’ time, our company will be having our annual meeting, where the three of us sit down and talk though plans and ideas for the next twelve months.
This means that Aaron (our US desk!) flies over to sunny England, and aside from a little socialising and catching up, we actually do some very productive work.
Taking a step back from the day to day routine is always a healthy thing, and when our three cultures come together face to face, the results are often more than a little amusing.
Aside from his misspellings, Aaron actually has an accent, which causes no end of amusement for my wife Karin and myself. Then there are the other issues.
Once he’s recovered from the fact that the whole country drives on the wrong side of the road, for some reason he seems to think that we all drive like madmen (or perhaps it’s just me) and that the roads are horrifically narrow.
Then there’s the currency issue. Oddly enough most of our shops (stores) don’t take dollars; we’re peculiar about that sort of thing.
But despite all of our differences, our two countries have far more in common than many people think. Even with the amusing accent, Aaron still enjoys his fish and chips just as much as any Brit. Even if he won’t put vinegar on the chips. Yet.
The Competitive Edge newsletter is a monthly in-depth look at the issues faced by independent software developers today.
If you’d like to comment on any of the information within the newsletter, please email Dave Collins directly atdave@softwarepromotions.com.
Common Sense Disclaimer: The advice, views, experiences, opinions and other information shared in the Competitive Edge newsletter are researched to the best of our ability. At the time of publication, we believe them to be correct.
SoftwarePromotions can not and will not guarantee that any of the information here is accurate, and/or will work for your company. In plain English, use this information at your own risk.
All information herein is offered “as is” and without warranty of any kind.
To the greatest extent allowed by law, SoftwarePromotions, nor its employees nor contributors, are not responsible for any loss, injury, or damage, direct or consequential, resulting from your choosing to use any of the information and/or advice presented here.
SoftwarePromotions
http://www.SoftwarePromotions.com
