Search Engine Optimization (SEO) is the process of improving the quality and boosting the quantity of website traffic, as well as growing your brand through information architecture and marketing strategies that get you found via organic search engine results.
With Google maintaining 92.24% of search engine market share worldwide, the focus of this article will likewise explain how Google Search works under the hood from an operational standpoint, as the search engine implicates itself with the responsibility of holding the keys to your potential success as a webmaster and content publisher. After attaining this operational knowledge, your SEO market reach strategy will subsequently extend itself towards understanding what users are searching for online, by using Google Trends to discover the words they are using to search, thus in turn reacting with answers to the questions they are querying, and creating the types of content they aspire to consume. Realizing all of this will allow you to better connect with users who are searching for the content, solutions, and services that you aim to offer online, as we at Solespire do for Luxury Real Estate Agents on The Pinnacle List.
Understanding the intentions of your audience is just one side of the coin for SEO though, as publishing content and delivering it in ways that best suit the operations of Google’s crawlers for indexing and ranking your content is vital on the flip side and of utmost importance to get started with organically amplifying your web traffic. This guide will help you master your understanding and appreciation of how Google Search works to maximize your online success as a web developer and content manager.
Crawling
Google surfs billions of pieces of content and evaluates an array of factors to determine which content is most likely to answer your query. Google computes this by discovering and cataloguing all available content on the Internet (web pages, PDFs, images, videos, etc.) via crawling and indexing, through which Googlebot visits newly published and updated pages that get added to the Google Search index.
Googlebot (generically known as a robot, bot, or spider) is a web crawler software program that discovers content. It uses a process, based on an ever-changing algorithm, to determine which sites to crawl, how often, and how many pages to fetch from each website. Specific functionalities of the algorithm have never been explicitly disclosed by Google, but SEO experts are always running their own processes to delineate the behavioural trends of Googlebot, as it relates to Google Search results.
The crawl process by Google commences with a list of web page URLs, generated from previous crawl processes, and is supercharged when any given website supplements bots with the augmentation of sitemap data, typically validated via an XML file provided by a webmaster. When Googlebot visits a web page, links from that page are gathered and added to its list of pages to crawl – effectively constructing a spider web of sorts, where new sites, updates on existing sites, and dead links are noted and used to update the Google index.
Primary and Secondary Crawlers
As user behaviour continues to evolve between desktop computers and mobile devices, Google has likewise established two sets of crawlers – a mobile crawler and a desktop crawler. Both crawlers are constantly simulating a user visiting web pages, so it’s imperative as a web developer to make sure your website is optimized for both platform types, exemplified by an iPhone for mobile or an iMac for desktop, among other suitable examples. It’s a fine line with tablets though, as Safari on iPadOS was designed using the desktop-class version of the WebKit browser engine developed by Apple, effectively mirroring its macOS counterpart, while many Android tablets still have Google Chrome running on mobile frameworks, despite Google’s heavy-handed efforts to consolidate Chrome across smartphone, tablet, and desktop types as far back as 2013; however, the fragmentation caused by software overlays, such as with the Samsung Galaxy series, has prevented that from ever becoming a true reality.
With all that considered, Google instituted a mobile-first indexing practice in 2019, which means Google predominantly uses the mobile version of your content for indexing and ranking your website and pages. Historically, the index primarily used the desktop version of a page’s content when deciphering the relevance of a page to a user’s query. Since the majority of users now access Google Search with a mobile device, Googlebot primarily crawls and indexes pages with their mobile crawler first. As such, some web designers opt to now draft a mobile-first website, with responsiveness considered from the standpoint of sizing up from the primary design via CSS or JavaScript, versus the traditional approach of coders using the viewport meta tag in HTML to control the width of the site in preparation for optimally sizing down, using CSS media queries. That being said, there is no wrong approach here, with the latter traditional approach still being the dominant approach because Googlebot does not take mobile-first or desktop-first design approaches into consideration. As long as your website is responsively optimized for mobile devices, your site has hastily earned permittance to be crawled by Googlebot.
When Google does not crawl a web page
- Pages that are blocked in robots.txt will not be crawled, but may still be indexed if hyperlinked to by another page, as Google can still infer the content of the page from a link that points to it, thus indexing the page without directly parsing its contents via Googlebot.
- Any pages not accessible by an anonymous user cannot be crawled for inclusion into Google Search results. Thus, any login or other authorization protections will prevent a page from being crawled.
- Duplicates of web pages that have already been crawled are crawled less frequently, and can even be blocked from appearing in Google Search results, which can hurt your site’s overall PageRank – one of the overarching algorithmic theories that rank web pages, which was developed in 1996 by Larry Page and Sergey Brin at Stanford University in 1996, as part of a research project, before the two launched Google in 1997.
Improve your crawling
These techniques will assist Google in discovering the pages you favour most to be crawled, indexed, and ranked:
- Submit a Sitemap.
- Submit crawl requests for individual pages, using Google Search Console.
- Use simple, clean, and concisely logical URL paths for your web pages. (e.g., https://www.travoh.com/articles/exploring-10-top-beaches-oahu-hawaii/)
- Provide clear and direct internal links within the site. (e.g., https://www.thepinnaclelist.com/listings/ with a site-wide menu in the header and an on-page navigational filter system that concentrates content in the specific section.)
- Use link insertions (often referred to as backlinks or inbound links, and sometimes linkbacks), which is a method for web authors to obtain notifications when other authors link to one of their blog posts – effectively assisting in constructing the quintessential spider web for crawlers. (e.g., https://www.thepinnaclelist.com/articles/how-much-does-small-plane-cost-we-detail-average-prices-buy-maintain/ with 3 link insertions.)
- Use robots.txt to notify Google of which pages you prefer Google to know of or crawl first, which can protect your server load, but should not be used as a method to block material from appearing in the Google index.
- Clearly identify canonical URL structures. (e.g., https://www.thepinnaclelist.com/listings/tpl53982-motu-tane-private-island-bora-bora-french-polynesia/ over https://www.thepinnaclelist.com/?p=8141)
- View your crawl and index coverage using Google’s Index Coverage Report.
- Make sure Google can access important resources attached to your web page, including images, CSS files, and scripts that may be necessary to render your page properly. Use the URL Inspection Tool to confirm that Google can properly access and render your page.
Indexing
Crawling and indexing are two distinct processes, which are commonly misunderstood in the SEO industry. Crawling means that Googlebot interprets the content and code of the page, while indexing means that the page becomes eligible to show up in Google Search results.
Between the events of crawling and indexing, Googlebot immediately determines if a page is a duplicate of another page or a preferred link, which is the canonical URL structure. If a page is deemed a duplicate, it will be crawled less frequently, or not at all. Any duplicates found that are meant to be alternate URLs of the same page, perhaps for mobile or desktop versions of the same content, or for different regional audiences like Canadian English or American English, will not receive a PageRank penalty, as long as you classify the web page with appropriate HTML attributes for the specific purpose of why it exists as a duplicate.
Ranking and Visibility
Ranking refers to the process used to determine where a particular piece of content should appear on Google’s modern Search Engine Results Page (SERP). Search visibility refers to how prominently web pages and content are displayed in search results. Highly visible content may appear right at the top of organic search results or even in a featured snippet, while less visible content may need to be found on page two and beyond.
Google’s SERP results differ in presentation from traditional organic text results. The most common SERP features include:
- Rich Snippets: An added visual layer to an existing result. (e.g., review stars for real estate ratings)
- Paid Results: Results bought by bidding on keywords. (e.g., AdWords)
- Universal Results: Concentrated content types. (e.g., image results, news results, featured snippets)
- Knowledge Graph: Data segregated as featured panels or boxes. (e.g., places, organizations, websites)
When a user inputs a search query, Google scours its index for matching pages and returns results it computationally believes are most relevant to the user, while considering meta details like the user’s location, language, and device type. For websites and web pages to appear higher in search results, Google also considers the user experience when choosing and ranking results, putting considerable merit on page speed and if the site is mobile-friendly.
Improve your site’s ranking in Google Search results
- If your content is meant to target users in specific locations or languages, tell Google your preferences.
- Make sure your pages load fast and are mobile-friendly.
- Follow the Google Webmaster Guidelines, which help you avoid common pitfalls, upon ensuring a good user experience that will improve your site’s overall ranking with Google.
- Consider implementing Search result features for your site, such as article cards.
The Google Search algorithm is constantly changing and as such, our SEO experts at Solespire task themselves to be on the leading edge of such changes with Multiplex, our innovative modern media stack built custom for our dynamically responsive content and advanced technologies on Solespire Media Sites that account for the newest web design practices, platform developments, and evolving content marketing strategies.
