I had a great conversation the other day with a friend who was asking about the importance of a sitemap. While a protocol exists, prompting an easy answer to most questions, it was proposed that following it didn’t really matter with Google owning search.
Fine opinion to take if you are the type to put all your eggs in one basket. What I encouraged, and hope you consider, is that when a convention, a standard, exists, follow it!
Where the conversation turned to debate was on the question of the location of the a sitemap (which, too, has a standard). Proposed, was that your robots.txt file points bots to the sitemap so its location doesn’t really matter.
While a sitemap in a unique location serves its purpose, and works just fine, I turned to an analogy (I like analogies) to defend the standard. Think of your sitemap as the host or hostess at your favorite restaurant. When you (the crawler) enter the restaurant, you expect to find that person right up front, smiling, behind a podium, with menus in hand; ready to show you to your table.
Now, keep yourself in the shoes of the search engine crawler and consider; how have you felt when you’ve entered a restaurant only to find that the host is missing from the podium? Probably not enough aggravation to turn you away but, I’m sure you’ve given pause, turned to your friends, and said, “well…”
Of course, as soon as the host appears, you’re satisfied and off to dinner but that moment, that infinitesimal moment, is a road block in an otherwise perfect experience.
What happens when that host isn’t there at all?
Without a sitemap, and hopefully you already realize this, your website is effectively a restaurant that says, “Please seat yourself.” Patrons are quite capable of seating themselves but, they pause, they debate, they point to open tables. For certain, your restaurant isn’t considered as fancy as the one next door.
So you realize you NEED a host, does it really matter where it lives? Why should a sitemap reside at https://yoursite.com/sitemap.xml? Perhaps more aggravating to me than the restaurant that doesn’t seat you (my expectation is usually previously set), is the one at which the podium is at the back of the restaurant, past the bar, through the tables. Max’s Opera Cafe at Stanford Mall does this (sorry Max); to get your name on a list for a table, you have to wade through the bar and the other customers waiting to be seated. Best advice I can offer? DON’T DO THAT!!
(Worse? They effectively do the same thing with their website; check it out, you have to wade through an intro before you get to the site).
Don’t make me clamor through the crowd just so I can say I want to get into the place! Don’t put the host somewhere other than where I expect it to be!! Why? Because when that search engine crawler comes crawling, it is going to look where it expects to find it, find nothing, and take a left turn at robots.txt before it gets in. Why put it through the hassle??
Now, I’m not one to put research behind this type of supposition. I haven’t tested whether or not it matters and someone could probably point out that it doesn’t. But I’m a patron at a restaurant and when a standard exists, stick with it, or I think a little less of you. Enough to walk away? No, not necessarily; but I probably won’t come back as much.
Final thought; of course this begs, what then is the robots.txt file if not the host? Doesn’t it point the direction? I suppose, it serves as the sign on the door; telling people whether or not you are open, hours of business, and what’s on the menu. The Yelp sticker is just showing off.
