Webmaster level: All

Our quality guidelines warn against running a site with thin or scraped content without adding substantial added value to the user. Recently, we’ve seen this behavior on many video sites, particularly in the adult industry, but also elsewhere. These sites display content provided by an affiliate program—the same content that is available across hundreds or even thousands of other sites.

If your site syndicates content that’s available elsewhere, a good question to ask is: “Does this site provide significant added benefits that would make a user want to visit this site in search results instead of the original source of the content?” If the answer is “No,” the site may frustrate searchers and violate our quality guidelines. As with any violation of our quality guidelines, we may take action, including removal from our index, in order to maintain the quality of our users’ search results. If you have any questions about our guidelines, you can ask them in our Webmaster Help Forum.

Posted by Chris Nelson, Search Quality Team

Webmaster level: Advanced

Over the years, Google has used different crawlers to crawl and index content for feature phones and smartphones. These mobile-specific crawlers have all been referred to as Googlebot-Mobile. However, feature phones and smartphones have considerably different device capabilities, and we've seen cases where a webmaster inadvertently blocked smartphone crawling or indexing when they really meant to block just feature phone crawling or indexing. This ambiguity made it impossible for Google to index smartphone content of some sites, or for Google to recognize that these sites are smartphone-optimized.

A new Googlebot for smartphones

To clarify the situation and to give webmasters greater control, we'll be retiring "Googlebot-Mobile" for smartphones as a user agent starting in 3-4 weeks' time. From then on, the user-agent for smartphones will identify itself simply as "Googlebot" but will still list "mobile" elsewhere in the user-agent string. Here are the new and old user-agents:

The new Googlebot for smartphones user-agent:
Mozilla/5.0 (iPhone; CPU iPhone OS 8_3 like Mac OS X) AppleWebKit/600.1.4 (KHTML, like Gecko) Version/8.0 Mobile/12F70 Safari/600.1.4 (compatible; Googlebot/2.1; +http://www.google.com/bot.html) (updated August 2015)
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The Googlebot-Mobile for smartphones user-agent we will be retiring soon:
Mozilla/5.0 (iPhone; CPU iPhone OS 6_0 like Mac OS X) AppleWebKit/536.26 (KHTML, like Gecko) Version/6.0 Mobile/10A5376e Safari/8536.25 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

This change affects only Googlebot-Mobile for smartphones. The user-agent of the regular Googlebot does not change, and the remaining two Googlebot-Mobile crawlers will continue to refer to feature phone devices in their user-agent strings; for reference, these are:

Regular Googlebot user-agent:
Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)

The two Googlebot-Mobile user-agents for feature phones:

• SAMSUNG-SGH-E250/1.0 Profile/MIDP-2.0 Configuration/CLDC-1.1 UP.Browser/6.2.3.3.c.1.101 (GUI) MMP/2.0 (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)
• DoCoMo/2.0 N905i(c100;TB;W24H16) (compatible; Googlebot-Mobile/2.1; +http://www.google.com/bot.html)

You can test your site using the Fetch as Google feature in Webmaster Tools, and you can see a full list of our existing crawlers in the Help Center.

Crawling and indexing

Please note this important implication of the user-agent update: The new Googlebot for smartphones crawler will follow robots.txt, robots meta tag, and HTTP header directives for Googlebot instead of Googlebot-Mobile. For example, when the new crawler is deployed, this robots.txt directive will block all crawling by the new Googlebot for smartphones user-agent, and also the regular Googlebot:

User-agent: Googlebot
Disallow: /

This robots.txt directive will block crawling by Google’s feature phone crawlers:

User-agent: Googlebot-Mobile
Disallow: /

Based on our internal analyses, this update affects less than 0.001% of URLs while giving webmasters greater control over the crawling and indexing of their content. As always, if you have any questions, you can:

• Read our recommendations for building smartphone-optimized sites
• Learn more about controlling Googlebot crawling and indexing
• Ask in our Webmaster help forums or visit one of our Webmaster Central office hours hangouts.

Posted by Zhijian He, Smartphone search engineer

Webmaster level: intermediate-advanced

In the past, we have seen occasional confusion by webmasters regarding how crawl errors on redirecting pages were shown in Webmaster Tools. It's time to make this a bit clearer and easier to diagnose! While it used to be that we would report the error on the original - redirecting - URL, we'll now show the error on the final URL - the one that actually returns the error code.

Let's look at an example:

URL A redirects to URL B, which in turn returns an error. The type of redirect, and type of error is unimportant here.

In the past, we would have reported the error observed at the end under URL A. Now, we'll instead report it as URL B. This makes it much easier to diagnose the crawl errors as they're shown in Webmaster Tools. Using tools like cURL or your favorite online server header checker, you can now easily confirm that this error is actually taking place on URL B.

This change may also be visible in the total error counts for some websites. For example, if your site is moving to a new domain, you'll only see these errors for the new domain (assuming the old domain redirects correctly), which might result in noticeable changes in the total error counts for those sites.

Note that this change only affects how these crawl errors are shown in Webmaster Tools. Also, remember that having crawl errors for URLs that should be returning errors (e.g. they don't exist) does not negatively affect the rest of the website's indexing or ranking (also as discussed on Google+).

We hope this change makes it a bit easier to track down crawl errors, and to clean up the accidental ones that you weren't aware of! If you have any questions, feel free to post here, or drop by in the Google Webmaster Help Forum.

Posted by John Mueller, Website Error Analyst

Cross-posted from the Inside AdSense blog.

We’ve heard from many publishers using WordPress that they’re looking for an easier way to work with Google products within the platform. Today, we’re excited to share the beta release of our official Google Publisher Plugin, which adds new functionality to publishers’ WordPress websites. If you own your own domain and power it with WordPress, this new plugin will give you access to a few Google services — and all within WordPress.

Please keep in mind that because this is a beta release, we’re still fine-tuning the plugin to make sure it works well on the many WordPress sites out there. We’d love for you to try it now and share your feedback on how it works for your site.

This first version of the Google Publisher Plugin currently supports two Google products:

• Google AdSense: Earn money by placing ads on your website. The plugin links your WordPress site to your AdSense account and makes it easier to place ads on your site -- without needing to manually modify any HTML code.
• Google Webmaster Tools: Webmaster Tools provides you with detailed reports about your pages' visibility on Google. The plugin allows you to verify your site on Webmaster Tools with just one click.

Visit the WordPress.org plugin directory to download the new plugin and give it a try. For more information about the plugin and how to use it, please visit our Help Center. We look forward to hearing your feedback!

Posted by Michael Smith - Product Manager

Webmaster level: intermediate

To help jump-start your year and make metrics for your site more actionable, we've updated one of the most popular features in Webmaster Tools: data in the search queries feature will no longer be rounded / bucketed. This change will become visible over the next few days.

The search queries feature gives insights into the searches that have at least one page from your website shown in the search results. It collects these "impressions" together with the times when users visited your site - the "clicks" - and displays these for the last 90 days.

Before and after:

We hope this makes it easier for you to see the finer details of how users are finding your website, and when they're clicking through. Should you have any questions, feel free to visit our help forum.

Posted by John Mueller, frequent searcher & Webmaster Trends Analyst, Google Switzerland

Webmaster Level: All

Search Queries in Webmaster Tools just became more cohesive for those who manage a mobile site on a separate URL from desktop, such as mobile on m.example.com and desktop on www. In Search Queries, when you view your m. site* and set Filters to “Mobile,” from Dec 31, 2013 onwards, you’ll now see:

• Queries where your m. pages appeared in search results for mobile browsers
• Queries where Google applied Skip Redirect. This means that, while search results displayed the desktop URL, the user was automatically directed to the corresponding m. version of the URL (thus saving the user from latency of a server-side redirect).

Skip Redirect information (impressions, clicks, etc.) calculated with mobile site.

Prior to this Search Queries improvement, Webmaster Tools reported Skip Redirect impressions with the desktop URL. Now we’ve consolidated information when Skip Redirect is triggered, so that impressions, clicks, and CTR are calculated solely with the verified m. site, making your mobile statistics more understandable.

Best practices if you have a separate m. site

Here are a few search-friendly recommendations for those publishing content on a separate m. site:

• Follow our advice on Building Smartphone-Optimized Websites
• On the desktop page, add a special link rel="alternate" tag pointing to the corresponding mobile URL. This helps Googlebot discover the location of your site's mobile pages.
• On the mobile page, add a link rel="canonical" tag pointing to the corresponding desktop URL.
• Use the HTTP Vary: User-Agent header if your servers automatically redirect users based on their user agent/device.
• Verify ownership of both the desktop (www) and mobile (m.) sites in Webmaster Tools for improved communication and troubleshooting information specific to each site.

* Be sure you've verified ownership for your mobile site!

Written by Maile Ohye, Developer Programs Tech Lead