Also, you can maintain a “sitemap” of your website that you submit to Google. The sitemap allows the spiders to reach even those URLs that are hidden away and are not easily accessible to spiders. In addition, you can “alert” Google in a variety of ways when you add or change content. You can remove outdated pages and prevent them from showing in Google’s index.

Google’s Webmaster tools also allows you see if there are crawling errors when Google visits your site. Meta descriptions, Titles and descriptions are all covered in it’s ambit. And finally, it lets you see which keywords were detected on your site and how your site ranks for various phrases – though I prefer Google Analytics  – more on that later.

For Google Webmaster Tools to work, you have to own the site or at least have read/write access to the files since it requires you to verify the site with either a meta tag or a specific HTML file.

It aims towards maximizing the download rate and in this process, it minimizes the overhead from parallelization to prevent repeated downloads of the same page.

The Crawling system requires a policy to avoid downloading the same page twice. The policy assigns the new URLs discovered during the crawling processes.

Dynamic assignment: Dynamic assignment helps a central server to assign new URLs to different crawlers in dynamic manners. This, in a way facilitates the central server to balance the load of each crawler.

The systems can efficiently add or remove down loader processes with the help of dynamic assignment. Since, in this case the central server becomes the bottleneck, it is very essential to transfer a large part of workload to the distributed crawling processes.

There are two main configurations of crawling architectures with dynamic assignments.

These configurations are well described by Shkapenyuk and suel in 2002.

a) A small crawler configuration: In this type of a configuration, there is a central DNS resolver and central queues per website and distributed down loaders.
b) A Larger crawler configuration: In this type of crawler configuration, both the DNS resolver and the queues are distributed.
c) Static assignment: There is a fixed rule in this type of policy. This defines how to assign new URLs to the crawlers.

In this policy, a hashing function can be utilized to transform URLs into a number that matches the index of the related crawling process.

It is necessary to exchange the URLs between crawling processes in batch. This reduces the overhead due to the exchange of URLs between crawling processes.

Three main properties constitute an effective assignment. They are: balancing property, contra-variance property and boldi et al.

Freshness: This indicates the accuracy of the local copy.

The formula depicted below defines the freshness of a page ‘S’ in the repository at time ‘t’.

Fs (t) = {1 If S is equal to the local copy at time t.

0 Otherwise.}

Age: indicates how outdated the local copy is.

The age of page ‘K’ in the repository, at time ‘B’ is defined by the formula given below.

A k (b) = {O If k is not modified at time b

B – modification time of b otherwise.}

Evolution of freshness and age in web crawling:

Edward G. Coffman used a different wording to define the objective of a web crawler that is equivalent to freshness. The analyzation resulted in a conclusion that a crawler must minimize the fraction of time pages remain outdated. They observed that the problem of web crawling can be modeled as a multiple-queue, single-server polling system, on which the web crawler is the server and the website are the queues.

The arrival of the customers are considered as page modifications and switchover times are the interval between page accesses to a single website. The waiting time for a customer in the polling system is equivalent to the average age for the web crawler.

Cho and Garcia Molina in 2003 studied two simple re-visiting policies as depicted below:

a) Uniform policy: The policy involves revisiting all pages in the collection with the same frequency, irrespective of their rates of change.
b) Proportional policy: The Policy includes the process of re-visiting more often the pages that change frequently.

Cho and gracia-Molina came out with an astounding result. They declared that the uniform policy out-performs the proportional policy in both a real web and stimulated web crawl. The explanation was based on the fact that frequent page changes wastes the crawler’s time by trying to re-crawl it too fast.

They emphasized on penalizing the elements that frequently change, to enhance freshness.

This is known as the optimal re-visiting policy that can neither be categorized under uniform policy nor proportional policy.

This is actually the best policy to keep the average freshness. The policy includes ignoring the pages that changes too often.

In this case the optimal is closer to the uniform policy. The revisiting policies here treat all pages as homogenous in terms of quality.

Politeness Policy: Koster found that using web robots is beneficial for a number of tasks. This accompanies a price to be paid for the general community.

The Costs include:

a) Network Resources: Robots require a considerable bandwidth.
b) Server overload: When the frequency of accesses to a given server is high.
c) Poorly written robots: can crash servers or routers.
d) Disrupted network and web servers: If too many users deplay personal robots.

Robots.txt protocol is a solution to the above-mentioned problems.