Get inside the head of a crawler

During this talk, we’ll try to become psychologists for a while and explore the mind of a crawler. Granted, finding out how they behave on your website and analysing server logs doesn’t sound particularly exciting at first, but just wait till you hear the case studies. You’ll walk away with a set of tools and concepts that will help you make your website much more accessible to crawlers and therefore improve your traffic.