Search Engine Facts You Need to Know

To achieve any goal, you first need to know everything there is to know about your goal and the variables that influence it. That way you can formulate a plan of attack that will work, and avoid time-wasting activities that will not. This applies to everything: running a business, waging a war, winning a race, and of course, marketing your web site. Information is power.

Today we will focus on some interesting facts about search engines and the Web. We'll see how we can use these facts to promote our sites through the search engines more effectively. Of course, there are many more ways you can market your web site, but the most effective both in results and in costs is getting included in the search engines and getting a good rank in searches for your products or services. That's because getting listed in a search engine is free, and if you are placed well, the traffic from an engine will literally feed you. Search engines are the most popular tools that web users use to find new information on the Net.

General Facts

Forrester Research estimates that there are 500 to 600 million pages on the Internet and that number is growing fast. However, the largest search engine, AltaVista, only has about 150 million pages indexed (about 27% of the Web), with Excite and Lycos at only about 50 million indexed (about 10% of the Web)! From September 1996 to September 1997, none of the search engines increased in size significantly, despite the fact that the web continued to grow! To a webmaster, these are shocking statistics! Why are relatively few pages are indexed? (1) The Web is growing faster than the engines can keep up with, and (2) many webmasters do not know how to design and submit their pages correctly. Getting and staying indexed well in a search engine takes a little more work than most people assume. You need a four-step approach.

The first thing you need to do is make sure all your web pages can be reached from your home page within three clicks. Most engines will only crawl three levels deep when indexing your site. Also, make sure all your pages have TITLE tags and META description and keyword tags, as most engines now use these. It is also highly advisable to have META category, language, and robot revisit tags, and ALT tags on all your images.

Don't just slap these into your pages. Put some thought into them. For example, the text in the TITLE tag for a particular page should start with a word that summarizes the entire page (a keyword). Say you have a page that mostly has information on vacations in Cancun, Mexico. Your TITLE tag should read something like 'Cancun vacations, tours, and travels in Mexico. Packages include diving...' The word 'Cancun' starts the sentence, and the rest of the sentence is made up of keywords that are related to the content of the page. This goes a long way toward getting you better rankings; same thing with the META tag text.

If you use frames on your site, make sure you use good NOFRAMES tags since not all major engines support frames. If you don't, your pages simply will not be indexed by those engines. If you use image maps, make sure you have a text links navigation bar somewhere on the same page, as not all major engines support image maps either. Quick note: the TITLE tag text should be at most 200 characters long, with the first 80 characters being the most important, as these are the ones most engines focus on in ranking and results display. Do not simply repeat keywords in the title tag. Make some grammatical sense out of the sentences but ensure that the keywords feature early and are not diluted by too many 'junk' words.

The second thing to do is submit only your home page and perhaps one other major page and then let the engines crawl your site. I will explain this in depth below. (The only exception is Infoseek. Infoseek does not crawl so you must submit every page on your site to it manually.)

Because the engines are so overwhelmed, you need a third step - you must monitor your submission and re-submit your home page every couple of weeks. The engine may have taken your submission but dropped it later (this happens a lot with Excite), gone to your site and found it unavailable at the time, or just not indexed your site due to a technical error on its part. Resubmitting and checking on your submission every two weeks will ensure that you will eventually get in and stay in the index.

Fourth, you need to get as many people linking to your site as possible. Visit related sites and ask for a link to your site. There is a trend by the engines to increasingly use link popularity and traffic as an indicator of relevancy. What this means is that the more people link to your page relative to your competitors' pages, the more highly you will rank on the engines. Not only will getting many incoming links get you a better rank on the engines, but it will also get you a lot of traffic (following links is the second most popular way people find new sites). Furthermore, on Excite, HotBot, and Lycos, link popularity also determines whether the engine will crawl deep into your site and index more pages or not. Do not ignore this fourth step, no matter how hard it sounds!

For the major engines, do not leave the submission process to automated programs and services. The major search engines are too important and the automated services sometimes do it wrong. You are only submitting the home page and one other major page to Excite, Lycos, AltaVista, Infoseek and HotBot - that is not much work to do manually every two weeks!

About Spamdexing

Because the search engines are so overwhelmed, they are coming up with more ways to make their job easier and weed out pages they feel are not worth indexing. One of the new developments is that most engines now insist or highly recommend that you only submit your home page to them and let the engine crawl through your site and index the pages it finds. If you decide to go against this recommendation and submit a whole bunch of pages through the online submission forms, you will risk being tagged a "spamdexer" (index spammer).

There is also an indication that engines like AltaVista give a higher ranking to crawled pages than submitted pages. So for your own interests, you want your pages crawled so that they have a higher score. Other engines like Excite will take the same amount of time to add your pages to their index whether you submit them manually or let it crawl to them from your home page. So not only will you be wasting your time submitting each and every page you have to Excite, but you will risk spamming that engine.

Conclusion: submit only your home page and one other major page and let the engines crawl your site. The only exception is Infoseek. Infoseek does not crawl so you must submit every page on your site to it manually. You can make a list of URLs to your pages and email that to Infoseek if you have more than 50 pages you wish to submit (see their submission page for more details).

There are a few other things to watch for to avoid having your pages excluded from the engines. The following make an engine tag a particular page as spam and therefore not index it. Make sure that none of your pages has any of these.

1. Keyword stuffing.
This is the repeated use of a word to increase its frequency on a page. Search engines have the ability to analyze a page and determine whether the frequency is above a "normal" level in proportion to the rest of the words in the document.

2. Invisible text.
Some webmasters stuff keywords at the bottom of a page and make their text color the same as that of the page background. This is also detectable by the engines.

3. Tiny text.
Same as invisible text but with tiny, illegible text.

4. Page redirects.
Some engines, especially Infoseek, do not like pages that take the user to another page without his or her intervention, e.g. using META refresh tags, cgi scripts, Java, JavaScript, or server side techniques. If you use redirection, it should have a delay of about 7 seconds.

5. META tags stuffing.
Do not repeat your keywords in the META tags more than 1 to 3 times, and do not use keywords that are unrelated to the content of your site.

6. Do not submit the same page more than once on the same day to the same search engine.

7. Do not submit virtually identical pages
i.e. do not simply duplicate a web page, give the copies different file names, and submit them all. That will be interpreted as an attempt to flood the engine.

Below are several useful facts and tips for each major search engine that you can use to improve your search engine marketing.

AltaVista Facts

Pages in index in millions: 150 Time it takes to index a submitted page: 1-2 days Time it takes to index crawled pages (may take longer than indicated): 1 day to 1 month

How to check if your page is on the index:
In the search box, type: '+url: yourcompany.com/yourpage.htm'. How to check how many pages link to your site: In the search box, type: 'link:yourcompany.com'.

You can narrow your search to a particular directory or page like: 'link:yourcompany.com/ourpage.htm'. To eliminate from the results all the pages within your own domain that link to each other, use the -url command: 'link:yourcompany.com -url:yourcompany.com'

Supports frame pages: Yes
Supports image maps: Yes

HotBot Facts

Pages in index in millions: 110 Time it takes to index a submitted page: 2 days to 2 weeks Time it takes to index crawled pages (may take longer than indicated): About 2 weeks

How to check if your page is on the index:
Select the advanced search options and enter your page's URL. How to check how many pages link to your site: In the search box, type: 'linkdomain:yourcompany.com'.

To eliminate from the results all the pages within your own domain that link to each other, use the -domain command like: 'linkdomain:yourcompany.com -domain:yourcompany.com'.

These methods get you all the pages linking to your domain. To find the links to only a particular page, enter your URL into the search box, then choose the "links to this URL" option.

Supports frame pages: No
Supports image maps: No

Infoseek Facts

Pages in index in millions: 75 Time it takes to index a submitted page: 1 day for pages submitted online, 7 days for email submissions.

Time it takes to index crawled pages (may take longer than indicated): Rarely spiders, if it does then 1 - 2 months

How to check if your page is on the index: In the search box, type: 'URL: http://www.yourcompany.com/page.htm'.

How to check how many pages link to your site: In the search box, type: 'link:yourcompany.com'.

You can narrow your search to a particular directory or page like: 'link:yourcompany.com/ourpage.htm'. To eliminate from the results all the pages within your own domain that link to each other, use the -url command like: 'link:yourcompany.com -url:yourcompany.com'

Supports frame pages: No
Supports image maps: Yes

Excite Facts

Pages in index in millions: 55 Time it takes to index a submitted page: About 2 weeks Time it takes to index crawled pages (may take longer than indicated): Up to 6 weeks

How to check if your page is on the index:
In the search box, type in the full URL of the page.

Supports frame pages: No
Supports image maps: No

Lycos Facts

Pages in index in millions: 50 Time it takes to index a submitted page: 2-4 weeks Time it takes to index crawled pages (may take longer than indicated): 2-4 weeks

How to check if your page is on the index: Not available.

How to check how many pages link to your site: N/A

Supports frame pages: No (limited)
Supports image maps: No

Yahoo! Facts

Yahoo is the most popular directory on the web. Many people have problems getting their site listed. A rough estimate is that only 1 out of every 10 submissions gets listed, if that. Moreover, it takes an estimated 4 to 15 weeks to be listed for those who actually get listed! Those who got listed had to resubmit their site an estimated 4 times over several weeks or months before getting listed (resubmitting often is spamming, by the way).

One thing is for sure - you must get into Yahoo! Yahoo actually brings some sites over 50% of their business. By the way, Yahoo now has an express submission service whereby you pay $199 for a response to your submission within 7 weeks. It doesn't guarantee that you will be listed with them, but at least you get to know within 7 days whether you are in or if not, why.

Watch out for our next article on "How to successfully get listed on Yahoo?"