What is Blog Content Scraping?

Blog content scraping is an act usually performed with scripts that extract content from numerous sources and pulls it into one site. It is so easy now that anyone can install a WordPress site, put a free or commercial theme, and install a few plugins that will go and scrape content from selected blogs, so it can be published on their site.

Why are they Stealing my Content?

Some of our users have asked us why are they stealing my content? The simple answer is because you are AWESOME. The truth is that these content scrapers have ulterior motives. Below are just few reasons why someone would scrape your content:

• Affiliate commission – There are some dirty affiliate marketers out there that just wants to exploit the system to make few extra bucks. They will use your content and other’s content to bring traffic to their site through search engine. These sites are usually targeted towards a specific niche, so they have related products that they are promoting.
• Lead Generation – Often we see lawyers and realtors doing this. They want to seem like industry leaders in their small communities. They do not have the bandwidth to produce quality content, so they go out and scrape content from other sources. Sometimes, they are not even aware of this because they are paying some scumbag $30/month to add content and help them get better SEO. We have encountered quite a few of these in the past.
• Advertising Revenue – Some folks just want to create a “hub” of knowledge. A one-stop-shop for users in a specific niche. If I had a penny for every time someone has done this with our content, then we would have a few hundred pennies. Often we notice that our site content is being scraped. The scraper always replies, I was doing this for the good of the community. Except the site is plastered with ads.

These are just a few reasons why someone would steal your content.

How to Catch Content Scrapers?

Catching content scrapers is a tedious task and can take up a lot of time. The are few ways that you can utilize to catch content scrapers.

Search Google with Your Post Titles

Yup that is as painful as it sounds. This method is probably not worth it specially if you are writing about a very popular topic.

Trackbacks

If you add internal links in your posts, you will notice a trackback if a site steals your content. This way is pretty much the scraper telling you that they are scraping your content. If you are using Akismet, then a lot of these trackbacks will show up in the SPAM folder. Again, this will only work if you have internal links in your posts.

Webmaster Tools

If you use google webmaster tools, then you are probably aware of the Links to your site page. If you look under “Traffic”, you will see a page that says Links to your site. Chances are your scrapers will be among the top ones there. They will have hundreds if not thousands of links to your pages (considering that you have internal links).

FeedBurner Uncommon Uses

If you have setup Feedburner for your WordPress blog, then you can see some uncommon uses. In the Analyze Tab under Feed Stats, you will see “Uncommon Uses”. There you will see a list of sites.

How to Deal with Content Scrapers

There are few approaches that people take when dealing with content scrapers. The Do Nothing Approach, Kill them all approach, Take Advantage of them approach.

The Do Nothing Approach

This is by far the easiest approach you can take. Usually the most popular bloggers would recommend this because it takes A LOT of time fighting the scrapers. This approach simply recommends that “instead of fighting them, spend your time producing even more quality content and having fun”. Now obviously if it is a well-known blog like Smashing Magazine, CSS-Tricks, Problogger, or others, then they do not have to worry about it. They are authority sites in Google’s eyes.

However during the Panda Update, we know some good sites got flagged as scrapers because google thought their scrapers were original content. So this approach is not always the best in our opinion.

Kill them all Approach

The exact opposite of the “Do Nothing Approach”. In this approach, you simply contact the scraper and ask them to take the content down. If they refuse to do so or simply do not reply to your requests, then you file a DMCA (Digital Millennium Copyright Act) with their host. In our experience, majority of the scraping websites do not have a contact form available. If they do, then utilize it. If they do not have the contact form, then you need to do a Whois Lookup.

You can see the contact info on the administrative contact. Usually the administrative, and technical contact is the same. The whois also shows the domain registrar. Most well-known web hosting companies and domain registrars have DMCA forms or emails. You can see that this specific person is with Hostgator because of their nameservers. HostGator has a form for DMCA complaints. If the nameserver is something like ns1.theirdomain.com, then you have to dig deeper by doing reverse IP lookups and searching for IPs.

You can also use a third party service for DMCA.com for takedowns.

Jeff Starr in his article suggest that you should block the bad guy’s IPs. Access your logs for their IP address, and then block it with something like this in your root .htaccess file:

Deny from 123.456.789

You can also redirect them to a dummy feed by doing something like this:

RewriteCond %{REMOTE_ADDR} 123\.456\.789\.
RewriteRule .* http://dummyfeed.com/feed [R,L]

You can get really creative here as Jeff suggests. Send them to really large text feeds full with Lorem Ipsum. You can send them some disgusting images of bad things. You can also send them right back to their own server causing an infinite loop which will crash their site.

The last approach that we take is to take Advantage of them.

How to Take Advantage of Content Scrapers

This is our approach of dealing with content scrapers, and it turns out quite well. It helps our SEO as well as help us make extra bucks. Majority of the scrapers use your RSS Feed to steal your content. So these are some of the things that you can do:

• Internal Linking – You need to interlink the CRAP out of your posts. With the Internal Linking Feature in WordPress 3.1, it is now easier than ever. When you have internal links in your article, it helps you increase pageviews and reduce bounce rate on your own site. Secondly, it gets you backlinks from the people who are stealing your content. Lastly, it allows you to steal their audience. If you are a talented blogger, then you understand the art of internal linking. You have to place your links on interesting keywords. Make it tempting for the user to click it. If you do that, then the scraper’s audience will too click on it. Just like that, you took a visitor from their site and brought them back to where they should have been in the first place.
• Auto Link Keywords with Affiliate Links – There are few plugins like Ninja Affiliate and SEO Smart Links that will automatically replace assigned keywords with affiliate links. For example: HostGator, StudioPress, MaxCDN, Gravity Forms << These all will be auto-replaced with affiliate links when this post goes live.
• Get Creative with RSS Footer – You can either use the RSS Footer or WordPress SEO by Yoast Plugin to add custom items to your RSS Footer. You can add just about anything you want here. We know some people who like to promote their own products to their RSS readers. So they will add banners. Guess what, now those banners will appear on these scraper’s website as well. In our case, we always add a little disclaimer at the bottom of our posts in our RSS feeds. It simply reads like “How to Put Your WordPress Site in Read Only State for Site Migrations and Maintenance is a post from: WPBeginner which is not allowed to be copied on other sites.” By doing this, we get a backlink to the original article from scraper’s site which lets google and other search engines know we are authority. It also lets their users know that the site is stealing our content. If you are good with codes, then you can totally get nuts. Such as adding related posts just for your RSS readers, and bunch of other stuff. Check out our guide to completely manipulating your WordPress RSS feed.

How You Can Reduce Blog Content Scraping and Possibly Prevent It

Considering if you take our approach of lots of internal linking, adding affiliate links, rss banners and such chances are that you will reduce content scraping to good measure. If you take Jeff Starr’s suggestion of redirecting content scrapers, that too will stop those scrapers. Aside from what we have shared above, there are a few other tricks that you can use.

Full vs. Summary RSS Feed

There has been a debate in the blogging community whether to have full RSS feed or summary RSS feed. We are not going to go into much details about that debate, however one of the PROS of having a Summary Only RSS feed is that you prevent content scraping. You can change the settings by going to your WordPress admin panel and going under Settings » Reading. Then change the setting For each article in a feed show: Summary.

Note: We have full feed because we care more about our RSS readers than the spammers.

Trackback SPAM

Trackbacks and Pingbacks definitely had great uses however, they are now constantly being abused. Often themes display trackbacks and pingbacks under or among the comments. This gives the spammer an incentive to scrape your site and send trackbacks. If you mistakenly approves it, then they get a backlink and mention from your site. Here is how you can disable Trackbacks on all future posts. Here is an article that will show you how to disable trackbacks and pings on existing WordPress posts as well.

Is Content Scraping Ever Good?

It can be. If you see that you are making money from the scraper’s site, then sure it can be. If you see a lot of traffic from a scraper’s site, then it can be. In most cases however, it is not. You should always try to get your content taken off. But you will realize as your blog gets larger, it is almost impossible to keep track of all content scrapers. We still send out DMCA complaints, however we know that there are tons of other sites that are stealing our content that we just cannot keep up with.

What are your thoughts? Do you use any other mechanics to prevent content scraping? Would love to hear your thoughts.

Beginner’s Guide to Preventing Blog Content Scraping in WordPress is a post from: WPBeginner which is not allowed to be copied on other sites.

First thing you have to do is install and activate the Code Freeze Plugin. Upon activation it shows a warning on all pages saying “Code Freeze is in effect. All Changes during this period will be lost”.

The plugin temporarily does the following while activated:

• Disables adding/editing/deleting new content, media, themes, etc.
• Disables installing/activating/deactivating/updating/deleting all plugins (except this one)
• Disables new comments/trackbacks on all content items
• Removes the QuickPress widget and notices to upgrade core or plugins
• Provides notice to dashboard users that any changes will be lost
• Provides generally “read only” access to the dashboard

Note: the only plugin you can deactivate is Code Freeze while the Code Freeze is activated.

This plugin is essential when you start the data migration process as well as when you are changing DNS. This plugin does not make any database or site changes. It also has no settings. It is definitely going in our MUST HAVE for moving a site.

Get Code Freeze Plugin.

How to Put Your WordPress Site in Read Only State for Site Migrations and Maintenance is a post from: WPBeginner which is not allowed to be copied on other sites.

Simply paste the following code in your theme’s functions.php file

function oddeven_post_class ( $classes ) {
   global $current_class;
   $classes[] = $current_class;
   $current_class = ($current_class == 'odd') ? 'even' : 'odd';
   return $classes;
}
add_filter ( 'post_class' , 'oddeven_post_class' );
global $current_class;
$current_class = 'odd';

Now simply style the odd/even class differently. You are good to go.

Source: Kevin Chard

How to Add Odd/Even Class to Your Post in WordPress Themes is a post from: WPBeginner which is not allowed to be copied on other sites.

Simply paste the following code anywhere you like in your theme files (such as footer.php).

<?php echo get_num_queries(); ?> queries in <?php timer_stop(1); ?> seconds.

And refresh the page. You will see the number of queries and the execution time.

How To Show Number of Queries and Page Load Time in WordPress is a post from: WPBeginner which is not allowed to be copied on other sites.

If you want to disable plugin updates, simply install and activate the plugin Disable WordPress plugin updates.

Now lets discuss why you should not disable plugin updates entirely. First of all, there is NO benefit of disabling plugin updates. It is very important that you keep your WordPress plugins up to date. If you don’t, then your blog could be susceptible to security vulnerabilities or performance issues. Some developers or consultants like this ability because they do not want the clients to upgrade a plugin (just in-case if it breaks the site). This is not a very good reason. You are potentially risking security, performance, and additional features all because of a fear that the site will break down due to an upgrade. It does not take that long to downgrade a plugin if the site breaks.

Another reason that developers disable plugin updates is if they have customized the core files. Again, this is not a good excuse to disable all plugin updates. You can use a plugin like WP Manage plugins which allows you to disable plugin updates for specific plugins.

There is no reason why you should disable plugin updates in WordPress for all plugins. If you have an argument for this, we would like to hear it in the comments section.

How to Disable Plugin Updates in WordPress and Why You Shouldn’t is a post from: WPBeginner which is not allowed to be copied on other sites.

First thing you need to do is install and activate the List Category Posts plugin. Once you have activated the plugin, you can simply use the shortcode anywhere in your post or page.

The shortcode is [catlist] which accepts numerous parameters. First and foremost it accepts the category name or ID parameter. Example would be like:

[catlist name=tutorials]

If you want multiple categories, its best to use the IDs. The shortcode would look like this:

[catlist id=7,24,32]

The shortcode accepts bunch of other parameters including tags, orderby, order, numberposts, date, author, excerpt, excludeposts, comments, thumbnail, custom fields, etc. The full list of parameters can be found in the Other Notes section of the plugin.

The plugin also has a widget for the sidebar that you can use by going to Appearance » Widgets.

Download List Category Posts

How to List Category Posts in a WordPress Post or Page with Shortcode is a post from: WPBeginner which is not allowed to be copied on other sites.

All you need to do is either open your site-specific plugin or your theme’s functions.php file and paste the following code:

// Add Theme Editor to Admin Bar (to save time!)
function admin_bar_theme_editor_option() {
	global $wp_admin_bar;
		if ( !is_super_admin() || !is_admin_bar_showing() )
		return;
		$wp_admin_bar->add_menu(
			array( 'id' => 'edit-theme',
			'title' => __('Edit Theme'),
                        'href' => admin_url( 'theme-editor.php')
		)
		);
	}

add_action( 'admin_bar_menu', 'admin_bar_theme_editor_option', 100 );

That is it. We found this code floating around on the WordCamp Nashville Hashtag. We tried it on WordPress 3.3.2 and it works. The code should work for other versions as well.

How to Add the Theme Editor in WordPress Admin Bar is a post from: WPBeginner which is not allowed to be copied on other sites.

How we use Coming Soon Pages

We have uses coming soon pages for three reasons.

1. Gauge User Interest – Sometimes we are just not sure if the idea is worth doing. We simply put up a coming soon page with email-optin, and see how many users are interested. Sure we would do some promotion for the site, but its worth spending a little bit of money to test things out before. You would lose a lot more money if you create a project just to find out that nobody cares about it
2. Create Hype – We have also used Coming Soon pages to create hype. Give the users a little info about what the product is and give them incentives to share the product. Incentives mostly are “Early Access to the site”. However, we have seen people giveaway iPads and other products as well.
3. Build an Email List – This is sort of similar to the first thing we mentioned. However, in this case we are pretty much sure about the project, so we also put links to our social profiles etc to start building a following before the launch. So the first day of launch, rather than having 0 people or 2 people liking your page, you have like few hundred.

Example of what we will Create

How to Create a Beautiful Coming Soon Page

First thing you need to do is get the SeedProd plugin. It is a paid plugin, but its worth getting it. Install and activate the plugin. Upon installing the plugin, you will see a new option that will appear in your Settings menu. It will be called Coming Soon Pro.

The general screen asks you to enter the API Key, check the status of the plugin, add google analytics code, and upload a favicon. So SeedProd also has a cool maintenance mode available as well which we will cover later in the article. Here is a screenshot of the general screen:

Until you build your coming soon page, you can keep it disabled. But it really doesn’t matter if it is a new site anyways. Next there is a Coming soon Options page. This is where you upload your logo, enter the headline and description, choose your email list options, select the option to display social profiles, add footer branding image and links.

Now where you notice it says Email List Options. SeedProd actually integrates with Aweber, MailChimp, ConstantContact, Campaign Monitor, Gravity Forms, FeedBurner and your database. The database option simply stores all emails collected in your database, and you can export it at anytime.

The next option is Coming Soon Style. Recently, the trend of having a coming soon page is to have a very cool background picture. Sometimes its not even related to the product. It could be some sort of abstract picture, and it could be of some sort of related picture (i.e product screenshot etc).

As you can see in this screen, you can have a background color as well as an image. We would highly recommend that you use a high quality image. You have the option to choose font colors. You have a limited number of color options which is one of the downside. We wish there was a color picker tool, so we can select whichever color we wanted. You have an option for the shadow box option, and you can select the opacity. You can see that this is pretty common in a lot of LaunchRock style coming soon pages. You have the option to customize the Link & Subscribe button color. You can select Headline and Body fonts. This plugin has all of Google Library fonts available. Lastly, there is a custom CSS box for those users who like to get their hands dirty. So it is a pretty handy set of options.

This plugin also has an incentive and sharing option. You can giveaway free ebooks, other digital products, coupons etc in exchange for user’s email.

As you can see, it also lets you display social buttons after the incentive. The user who gives you their email is much more likely to share your stuff than an average user.

There is a screen where you enter your Social Profile URLs. It currently allows for Twitter, Facebook, Google+, LinkedIn, and Pinterest. Entering URL part is pretty straight forward.

Maintenance Mode Option

In the past we have shown you how to create maintenance websites in WordPress with plugins and without plugins. SeedProd also has the ability to let you create cool looking maintenance pages in few minutes. The trick is to have a very good looking background image.

Advanced Settings

This plugin has a few handy advanced settings. First and most important is the SEO settings. You can enter the site’s SEO title and Description. If you are using SeedProd maintenance mode, then there is an option for Client View URL. This special url would allow the client to visit the actual site rather than seeing a coming soon or maintenance page. It also has the option to exclude specific URLs. For example, if you are working on revamping the main site, but you still have your blog up. You can exclude a specific URL such as yoursite.com/blog/. This would allow you to redirect your users to the blog page for updates.

The plugin has the ability for you to add Header and Footer Scripts. Which is nice to have, but we can’t see any use for it just yet. Last, but certainly not the least, it also has the option for Import / Export. This will come in handy, if you like to follow specific settings.

You have already seen the final results of the product. We would highly recommend getting SeedProd if you are looking to build a beautiful coming soon page without the coding skills. You can literally put up a coming soon page in less than 10 minutes granted that you have the pictures and everything.

Get SeedProd Now

How to Create Beautiful Coming Soon Pages in WordPress with SeedProd is a post from: WPBeginner which is not allowed to be copied on other sites.

Three external libraries included WordPress received security updates including Plupload, SWFUpload, and SWFObject. This release also addresses the Limited privilege escalation issue, cross-site scripting vulnerability when making URLs clickable, and cross-site scripting vulnerabilities in redirects after posting comments in older browsers. Five other bugs were also fixed in WordPress 3.3.2. You can look at the full changelog for more details.

What are you waiting for? Upgrade your sites NOW!

WordPress 3.3.2 (Security Update) – Upgrade Now is a post from: WPBeginner which is not allowed to be copied on other sites.

First thing you should try is open your .htaccess file (you can actually do this from WordPress SEO plugin > Edit Files option) and simply add the following code in there:

# WordPress SEO - XML Sitemap Rewrite Fix
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^sitemap_index\.xml$ /index.php?sitemap=1 [L]
RewriteRule ^([^/]+?)-sitemap([0-9]+)?\.xml$ /index.php?sitemap=$1&sitemap_n=$2 [L]
</IfModule>
# END WordPress SEO - XML Sitemap Rewrite Fix

For most people this fixed the issue. Someone reported that while this code fixed the issue, the page was still sending the 404 response header which meant that Google Webmasters tool couldn’t find it. Well in our client’s case, the above solution did NOT fix the issue.

While not preferred, we had no choice but to edit the core plugin files to fix the issue. As suggested by hadjedj.vincent, we edited the class-sitemaps.php located in “/wp-content/plugins/wordpress-seo/inc/class-sitemaps.php”.

You need to look at the function init() code and make that section look like this:

/**
	 * Initialize sitemaps. Add sitemap rewrite rules and query var
	 */
	function init() {
		global $wp_rewrite;
		$GLOBALS['wp']->add_query_var( 'sitemap' );
		$GLOBALS['wp']->add_query_var( 'sitemap_n' );
		add_rewrite_rule( 'sitemap_index\.xml$', 'index.php?sitemap=1', 'top' );
		add_rewrite_rule( '([^/]+?)-sitemap([0-9]+)?\.xml$', 'index.php?sitemap=$matches[1]&sitemap_n=$matches[2]', 'top' );
		$wp_rewrite->flush_rules();
	}

Basically what we are doing is adding the global $wp_rewrite; before everything that is already there and adding flush_rules after the code. By adding this in the plugin, it seems to fix the 404 issue on our client’s website. We have no idea why Yoast is not doing this by default, but it seems that quite a few users are having this sitemap 404 issue.

If you were looking for this issue, we hope that this fix has helped you.

How to Fix Yoast’s WordPress SEO Sitemap 404 Error is a post from: WPBeginner which is not allowed to be copied on other sites.