Google and the West Wind Message Board

February 10, 2004 @ 10:42 pm

I recently (what the hell took me so long) figured out that Google is not indexing the Message Board on this Web Site, or indexing it very incompletely anyway.

Actually this was one of the main reason I started using this WebLog instead of the Message Board to push out some random ramblings and code content, so I can actually find it again in the future when I might need it and share with some people. Judging from the amount of incoming developer search traffic hitting this Web Log, it's providing a good service.

But now I'm wondering what is it that's keeping Google and other search engines out of the Message Board. There are a few things that come to mind:

Frames based
The Message Board uses frames and I have a feeling the issue is that Google doesn't easily jump into frames pages. Then again I've found plenty of other contents in frames based applications as as well, so that alone can't be it. Further, once I found out about this I put a direct link onto the home page to the Message List page which jumps into the MB without frames from which a crawler could certainly get at the content.
Long lists of Links
The entry point to the Message Board is basically this list of current links, which is just about all links with no other content. I think Google uses some rules to not access pages that are geared at 'link spamming' a crawler. I think this may be more the reason, because I seem to not run into very many Message Board links from other sites either with a few exceptions. For example, I don't ever recalling seeing a link on www.asp.net displayed in a search.
Robots.txt
Yes I checked it and made sure that it doesn't exclude crawlers .
Link Naming
Urls on the Message Board have a .wws extension and usually include a querystring with message ids. Search engines tend to me more partial to 'static' or seemingly static content at a particular page, rather than a generated page with a dynamic URL.

Anybody else have any ideas what rules would keep out crawlers?

I suppose it's not that big of a deal, since the Message Board includes its own search that will turn up what you're looking for, but it'd be very useful to understand what is causing the exclusion.

Feedback for this Weblog Entry

re: Google and the West Wind Message Board
by Paul Nicholls February 15, 2004 @ 3:59 pm

I don't think Google are penalising you, because your main frame pagerank is 6 out of 10. However, I do believe Google tries to avoid spidering messageboards when it can (or at least spider them less).

The actual messageboard frame has a pagerank of 0; direct links to it would probably help a bit.

Your messageboard link at the top of the news frame actually points to the outside frameset, which results in loading the whole frameset inside that news frame - did you mean to do that?

Other things to consider - you have a dropdown for forums - links would be better here. Also, link back to a main forum list from each topic, as well as the forum the topic belongs to.

Hope some of that helps, good luck!

re: Google and the West Wind Message Board
by Rick Strahl March 02, 2004 @ 1:08 am

After reading through some related discussions I think I know why it's not getting spidered: Most search engines ignore dynamic urls that include query strings or at least complex querystrings. This is why most WebLogs and commerical message boards try to use vanity Urls that break up messages a different way. I'm considering this for the message board at least for message posts. It sure would be nice if the content could be searchable by Google since there are many gems in the discussions...