How Does SEO Work? Understanding Crawlers

In order to optimize your placement on Google searches, it's useful to know how search engines find site pages online and rank them for certain keyword searches. Here's a (super short!) explanation of how it works!

Interested in going deep on this topic? Check out a much more in depth explanation here​

Crawlers: How Search Engines Find Your Website

Before a search engine can rank the results of a search, it needs know what is out there. It sounds kind of creepy, but search engines have a catalog of everything that they can find on the internet using their crawler algorithm. These are like little robots that follow links throughout the web to "index" the content that's out there. In order to show up in search results, you need to get crawled first.

Here's how to make sure your whole site is getting crawled:

• Be patient! It takes a couple days to register with search engines

• Make sure all pages on your site link to another page so the crawlers can find them

• Simplify your navigation

• Make sure content isn't hidden behind login forms, search forms, or in images

• Have an existing website link to yours

If you're using a site template like Squarespace or Shopify, they will provide a sitemap for Google to follow, so it should maximize your chances of being crawled thoroughly.

Once your pages are indexed, the search engine will store the content and return it when somebody searches something related to this content (more on this process later!) As you continue to update your site, search engines will periodically re-crawl your site and update the content accordingly. Keep this in mind as you update or change the pages on your site. For example, if you are closing down a page, make sure to deactivate it or delete it to prevent it from appearing in search results as a dead/outdated link.

Now that we understand how search engines find and index content, we can discuss how results get ranked when you search!