You would have definitely come across Yoast SEO plugin if you own a blog. Yoast SEO is a widely used plugin in WordPress community. They have a massive user base and have been downloaded more than 1 million times.
Yoast SEO plugin has been used by many big players in the industry like SearchEngineLand, Elegant themes, W3Layouts etc. which shows they have a good reputation in the industry.
So even we were using Yoast SEO plugin to optimize our websites and make it as search engine-friendly as possible. They had a wide range of features which compelled us to use the plugin in all our websites
- Yoast SEO plugin allows us to add meta title and description for each post and based on the content added, it analyzes your content for the target keyword.
- You can easily hide or show your preferred content to search engines. Suppose you want to create a landing page and you don’t want that to appear in search results then you can easily hide the page directly from Yoast plugin.
- Editing your robots and .htaccess is made easy through Yoast plugin. Earlier these files could be accessed only through FTP as they were available in the root directory.
- Redirect managers to easily set up redirects in your site.
We got hit by Google Penalty and traffic dropped by 75%
Just when we thought everything was going well, we were in for a big shock when we got hit by Google penalty in the month of March that dropped the traffic of one of our main websites by 75% overnight. To give you the precise numbers the average daily traffic fell from 7000 to 1000 which was huge and the worst part was we had no clue on what exactly happened
During our analysis, we found that it was caused by a wrong setting in Yoast
We spoke to many SEO experts on this issue but no one had any answers which was really frustrating. We later found that, just like us, lots of people saw a drop in their traffic from March 7th to March 20th.
We tried the following things to fix this unknown problem
- Removed all the Ads from all the pages.
- Removed all the plugins which were not absolutely necessary.
- Removed most of the affiliate links (except some links in the review posts).
- Fixed 404 errors with proper 301 redirect links.
- Updated Robot.txt file.
- Enabled AMP on the website.
- Un-indexed pages which were not necessary.
- Fixed Yoast SEO settings.
Google webmasters should be the first thing that you need to look into when there is a drop in your traffic.
To our surprise when we looked into Google Webmaster, we found that the number of index pages increased to 500+ when we had less than 100 pages on our website.
When we analyzed this issue further, we found out that due to a wrong setting in the Yoast Plugin some of the images were getting indexed as pages.
As a result, lots of images on our website were indexed on Google as separate pages, attracting the Google penalty and ultimately affecting our SEO.
How did we recover our traffic?
When we saw that there were more than 500 indexed pages, we changed our Yoast media and attachment URL setting to YES
But as tons of images were already indexed as pages on Google, we had to submit URL removal manually for all the pages.
You can find that setting under Google Webmaster -> Google Index -> Remove URL
We felt this step was the most effective solution as it helped us recover traffic when we saw the indexed page numbers reduce to 200 from 500. However, we still couldn’t reduce the indexed page number to the original value since actual de-indexing of page takes time and depends on Google
Surprised to know that the wrong setting was due to a bug in Yoast SEO plugin
To give you a brief background on this issue, let’s understand how WordPress works with images. When you upload any image in WordPress, it automatically creates a separate page for every image that you upload on your website.
Since these pages don’t add any value to the user they are considered to be bad for SEO. Yoast SEO plugin has a very cool feature where it lets you redirect these image pages to the post page.
But when you would have updated your Yoast Plugin to Yoast 7.0/Yoast 7.0.2, the settings for redirecting the image pages got affected due to the bug in the plugin. As a result, “Media and attachment url” setting was set to NO.
Now, lot of your image pages were indexed in Google as separate pages which is really bad for your SEO.
Being such a popular and widely used SEO plugin, it was highly irresponsible from Yoast to let this bug slip through their updated version. We expect them to have a better testing process to make sure that the updates don’t have any negative impact on SEO as so many people get affected in terms of their rankings and monetization.
This is what one Yoast user had to say about this issue and how the drop in traffic affected her business
What did Yoast founder have to say about this bug
Yoast accepted their bug and its founder offered a public apology to all the users who were affected by the rankings. This is what Joost De Valk had to say about the issue in his blog post related to yoast SEO bug
What to do if you got hit by the same issue
If you are someone who is affected by this issue then we would advise you to change your Yoast media and attachment URL setting to YES immediately.
Log in to your WordPress website, on the left side of the menu select YOAST SEO icon. From the sub menu items displayed select “Search Appearance”
You will be taken to the dashboard. Click on Media tab and make sure to set your Media & Attachment URLs to “Yes” as shown below.
The solution provided by Yoast SEO Plugin
Yoast plugin through their blog post have recommend us to first change Media & Attachment URLs to YES which we already discussed in the above section. But the problem with this method is it could take months for Google to crawl those pages and remove those URLs.
Because of this reason you need to download another plugin provided by Yoast known as “Yoast SEO Search Index Purge” and activate the plugin in order to repair the bug completely
Let’s look at what this plugin does….
Yoast SEO Search Index Purge Plugin makes all the attachment URLs or the image pages in your website return to 410 status code, helping you remove these pages out of search results as soon as possible.
Next, XML sitemap will be created containing all the attachment URLs (with recent post modified date). This makes Google crawl all these attachment URLs again.
Yoast believes that after six months the attachment URLs or image pages would be removed from your search results page
Then, you can remove Yoast SEO Search Index Purge Plugin and just keep the redirect setting of the attachment URLs set to “Yes”.
The installation process is very simple. You just need to download and activate the Yoast SEO Search Index Purge plugin in your WordPress site.
Once the plugin is activated just navigate to Yoast SEO > Search Appearances and verify the media tab displayed. Make sure the following text “These settings are being overridden by the Search Index Purge Plugin” is displayed in Media Tab
Lessons learned from this experience
As we get most of the traffic from SEO, we have decided to be extra careful and monitor our site for any on-page SEO issues after updating our important plugins. Please share your experience in the comments section if you have faced this issue. We would love to know your thoughts