Site using PBNs and blocking crawlers moving up, how to compete?

[Darth]

I run a white hat site that does well in the SERPs for its niche. Over the last 6 months a very content-thin site has started to creep up the rankings and out-rank me and other big authoritative sites for certain terms.

I look at this sites backlinks with the usual tools and don't see anything special though, nothing that should be causing it to rank to highly (by a long shot). Also, the on-site SEO is nothing special. Like based on what I see it should be back in page 5+, not the top 3 of page 1!

So I've concluded they are using PBNs and the PBNs are blocking crawlers so I can't see them. Is there anything I can do to compete more effectively with this site?

 

[Jitsiereveld]

I look at this sites backlinks with the usual tools and don't see anything special though

How many levels deep did you use those tools? Still unable to see anything?

 

[Darth]

How many levels deep did you use those tools? Still unable to see anything?

I meant ahrefs, moz, majestic, openlinkprofiler etc are the tools I've used to try and see their backlinks

 

[Jitsiereveld]

Just checking how deep you looked into their backlinks' backlinks. But using the right tools, I'm sure you know what to look for.

Edit: Unfortunately, I'm not the expert on what you can do to compete more effectively with this site? Sorry I couldn't be more help.

 

[Nat]

You could try building some more / better links or optimizing on-page. If another site is ranking better than you with poor content and paid links, and you're staying 100% white hat, what are your other options?

 

[Darth]

You could try building some more / better links or optimizing on-page. If another site is ranking better than you with poor content and paid links, and you're staying 100% white hat, what are your other options?

Thanks that is that I have started doing, but is there any possible way to see their PBN links?

 

[Nat]

Thanks that is that I have started doing, but is there any possible way to see their PBN links?

I'm no SEO expert. But, if they blocked crawlers I don't think you're going to be able to find out what the PBN links look like unless you have access to a crawler that isn't blocked in their robots / .htaccess. If you think they used some brand anchors, you can always use some basic Googlefu skills and search for "site name," "author name," and "sitename.com." You might find a few of the sites... and could then try to reverse engineer / find a footprint / provider... but any half decent PBN isn't going to have obvious footprints.

 

[eliquid]

I'd tell you to do nothing at all.

Others might say Im crazy.

Wanna know what I've seen kill many-a-sites from the 17+ years I've been doing this?

Trying to keep up with someone that's prob on the verge of getting penalized anyways.

 

[Ryuzaki]

I agree with eliquid. Don't do anything fishy in order to regain your spots. I agree with Nat. Do more white hat stuff. They will tank soon enough. Be in a position to retake your spots and even gain more over competitors.

Also, while that ranking is being manipulated, go out and take other ones over so you won't even care when things like this happen. Just wait it out and grow other pages, which will strengthen your entire domain. If you hyperfocus on this, you'll goof up and go insane.

 

[CCarter]

Actually your competitor is doing everything right. By using the htaccess/nginx bad crawler/robots scripts to block unwanted bots, which we provide, throughout all his PBNs and his sites, his competitors can't see anything using those 3rd party tools. That's sort of the point. The Opps can't catch him lackin - you gotta keep the Opps blind to your activities.

He's not able to look up his own stuff, but being the invisible man gives him more power. But here is the thing to remember, you can still look up the other 8 competitors in the top 10, get the gauge for the metrics they've got, moz DA/PA, majestic TF/CF, and ahrefs DR, then once you get the averages, lows and highs, you'll know what you need to increase yours by to get back your position.

But here is the real question, even if you found his PBNs, what were you going to do? You'd have to get the metrics on all his operation and I guess get the lows/high and see what it takes for you to regain I guess. But other than that what else was there to do once you seen his PBNs?

 

[Darth]

I'm no SEO expert. But, if they blocked crawlers I don't think you're going to be able to find out what the PBN links look like unless you have access to a crawler that isn't blocked in their robots / .htaccess. If you think they used some brand anchors, you can always use some basic Googlefu skills and search for "site name," "author name," and "sitename.com." You might find a few of the sites... and could then try to reverse engineer / find a footprint / provider... but any half decent PBN isn't going to have obvious footprints.

Thanks I tried doing the Google searches but nothing turned up.

I'd tell you to do nothing at all.

Others might say Im crazy.

Wanna know what I've seen kill many-a-sites from the 17+ years I've been doing this?

Trying to keep up with someone that's prob on the verge of getting penalized anyways.

Sensible advice thanks.

Actually your competitor is doing everything right. By using the htaccess/nginx bad crawler/robots scripts to block unwanted bots, which we provide, throughout all his PBNs and his sites, his competitors can't see anything using those 3rd party tools. That's sort of the point. The Opps can't catch him lackin - you gotta keep the Opps blind to your activities.

He's not able to look up his own stuff, but being the invisible man gives him more power. But here is the thing to remember, you can still look up the other 8 competitors in the top 10, get the gauge for the metrics they've got, moz DA/PA, majestic TF/CF, and ahrefs DR, then once you get the averages, lows and highs, you'll know what you need to increase yours by to get back your position.

But here is the real question, even if you found his PBNs, what were you going to do? You'd have to get the metrics on all his operation and I guess get the lows/high and see what it takes for you to regain I guess. But other than that what else was there to do once you seen his PBNs?

Is there anything negative that can happen to my own rankings by blocking that stuff? I assume the only effect will be that I just don't see my own metrics on Majestic etc? If I block now, how long until my site is removed from tools like Majestic? I want to find his network to confirm that I am not missing something very obvious, and just to see what kind of PBN/links they have. They must be some pretty good links to be beating out mine and the other sites - domains with like 85 DA (not mine).

 

[Nat]

But here is the real question, even if you found his PBNs, what were you going to do? You'd have to get the metrics on all his operation and I guess get the lows/high and see what it takes for you to regain I guess. But other than that what else was there to do once you seen his PBNs?

Yeah ^^ this is the point I was trying to make.

Is there anything negative that can happen to my own rankings by blocking that stuff? I assume the only effect will be that I just don't see my own metrics on Majestic etc? If I block now, how long until my site is removed from tools like Majestic? I want to find his network to confirm that I am not missing something very obvious, and just to see what kind of PBN/links they have. They must be some pretty good links to be beating out mine and the other sites - domains with like 85 DA (not mine).

Blocking bots won't hurt you - https://www.buildersociety.com/threads/block-unwanted-bots-on-apache-nginx-constantly-updated.1898/

 

[Darth]

OK this is weird. I looked at their robots.txt and this is it:

User-agent: *
Disallow: /wp-admin/
Allow: /wp-admin/admin-ajax.php
​

Their .htaccess:

# BEGIN WordPress
<IfModule mod_rewrite.c>
RewriteEngine On
RewriteBase /
RewriteRule ^index\.php$ - [L]
RewriteCond %{REQUEST_FILENAME} !-f
RewriteCond %{REQUEST_FILENAME} !-d
RewriteRule . /index.php [L]
</IfModule>

# END WordPress
​

Also their server is Apache so no hiding blocking rules in the nginx conf. So they aren't blocking anything? In which case their links are crap and they are still ranking high. But their on-page SEO is average too - as an example they haven't even removed the "Just another Wordpress site" alt text from the logo.

 

[CCarter]

@Darth, You are mis-understanding the techniques. It's the PBN websites that have the block for ahrefs and other 3rd party tools. By not allowing ahrefs to crawl the PBN site it doesn't allow ahrefs to report OUTBOUND links to the money site. When you apply the blocking technique it blocks outbound links from showing up in the 3rd party tools. That's why when you look up the money site it looks like there are no backlinks. The block is done on the PBN level.

If you were to implement this on your money site it would only not show website you are outbound linking to. It will still show sites that are linking to your money site. You would have to implement the technique on the sites outbound linking TO YOUR money site.

 

[Darth]

@Darth, You are mis-understanding the techniques. It's the PBN websites that have the block for ahrefs and other 3rd party tools. By not allowing ahrefs to crawl the PBN site it doesn't allow ahrefs to report OUTBOUND links to the money site. When you apply the blocking technique it blocks outbound links from showing up in the 3rd party tools. That's why when you look up the money site it looks like there are no backlinks. The block is done on the PBN level.

If you were to implement this on your money site it would only not show website you are outbound linking to. It will still show sites that are linking to your money site. You would have to implement the technique on the sites outbound linking TO YOUR money site.

Of course! Major mind melt down moment

 

[lion1978]

But here is the real question, even if you found his PBNs, what were you going to do?

I guess report paid links https://support.google.com/webmasters/answer/93713?hl=en
https://support.google.com/webmasters/answer/93713?hl=en
https://www.google.com/webmasters/tools/paidlinks
https://www.google.com/webmasters/tools/paidlinks
I know it's considered a shitty and by some blackhat move to do. But it works very often, mainly becouse to many "seo's" don't know how to make PBN's that actually looks and feels like real sites, both in content, traffic and ads.

 

[Darth]

I guess report paid links https://support.google.com/webmasters/answer/93713?hl=en
https://www.google.com/webmasters/tools/paidlinks
I know it's considered a shitty and by some blackhat move to do. But it works very often, mainly becouse to many "seo's" don't know how to make PBN's that actually looks and feels like real sites, both in content, traffic and ads.

yeah im not sure how id feel about doing that. I have gone grey hat on some of my sites before and karma can be a bitch

 

[lion1978]

karma can be a bitch

Indeed she can and that is just one reason to always keep a copy of your site somwhere.
then if it was ever to happen to you, a competitor reporting you that is, there is an easy way to fairly quickly regain your rankings or even boost them up using a new domain, the funnny thing is this "trick" is so white hat that almost only blackhats know it exist.

 

[knownerror]

As I've said before. Those that rat out their competitors are just that, rats. Don't be a dick and ruin the game for everyone.

 

[Michael]

It's a dirty game out there, but it's also 2017. If people cannot hide their PBN links, or poorly disguise their links as non-pbn links, would you guys think they kind of deserve to be reported?

I'm not one to play that dirty, but that's literally low-hanging fruit for SEOers, no?

Karma definetly plays in my head on this one.

 

[CCarter]

I know it's considered a shitty and by some blackhat move to do. But it works very often, mainly becouse to many "seo's" don't know how to make PBN's that actually looks and feels like real sites, both in content, traffic and ads.

 

[Michael]

As I've said before. Those that rat out their competitors are just that, rats. Don't be a dick and ruin the game for everyone.

True, but then again, literally anyone can do it so being a "rat" or a "snitch" in this case may not be valid.

 

[lion1978]

I know it is

 

[knownerror]

True, but then again, literally anyone can do it so being a "rat" or a "snitch" in this case may not be valid.

No. Only those that lack the skills and intelligence to compete with a competitor rat them out instead. If you can't rank ahead of someone and need to inform G of their linking activities, just get out.

 

[Michael]

No. Only those that lack the skills and intelligence to compete with a competitor rat them out instead. If you can't rank ahead of someone and need to inform G of their linking activities, just get out.

So youre telling me the big dog markets for the big companies don't employ dirty tricks because "they lack skills and intelligence"?