Rewrite URL Parameters to Directories for Search Indexing?

[animalstyle]

I am working with a 2 variable input type that influences the content displayed on the page. The goal is to create pages that are indexed and show in SERP's. For example, I want people to be able to search video game types by console and the resulting page displays the relevant info:

IE: RPGs for Playstation

Building this out with actual pages would be a really large task, and I don't think its the way to handle it ie:

www.site.com/find/rpgs/playstation/

I've worked on a solution to handle this with URL parameters like:

site.com/find/?type=rpgs&console=playstation

I know Google CAN handle indexing these URLs and you can suggest that the page information changes based on the type/console variables - but it doesn't seem ideal for them or a user.

Would the ideal way to handle this situation be to re-write the URLs based on the URL parameters into a directory structure? Would the URL parameters still be passed?

 

[Ryuzaki]

As far back as 2007, Matt Cutts was talking about how Google does not want to index search pages:

• https://www.mattcutts.com/blog/search-results-in-search-results/

Today, we have Panda taking care of this for us, punishing us for auto-generated content that's largely a duplicate of other pages. Like RPG's on Xbox and RPG's on Playstation are going to be 95% the same posts showing up.

Will you have so many different variations that a search function is necessary instead of using custom loops to pull the relevant content automatically for you? Or is this one of those eCommerce style sidebar filters with checkboxes?

If there's not going to be 1000's of these pages, and maybe less than 50 to 100, I think you'll be fine, but I'd might add a static paragraph at the top of each so there's something unique on each page. I'd suggest setting up a custom page template that duplicates the results of this search. Then I'd create a static URL for each search result and 301 it, and tell this static URL page to use that custom page template. Then it can still automatically update for you but appear more like a Blog Loop or Category style page.

I'm not even sure that's ideal. I'm on my first coffee of the day, my brain isn't working yet.

 

[animalstyle]

The example I gave was a bad one. Let me try again:

I'm creating listings of locations in the US. I'll have pages aimed at each state, and likely major cities. So just there I have 50 state pages + 100's of city pages for the 'all category' of locations. I'd like to then split it down into location categories and each time I'd do that, it would multiply the number of state/city pages. There could be 1000's of them as I build out categories.

Does that make sense?

The idea is trying to create relevant pages for terms like 'location category' in 'city/state'. IE Pizza in Chicago that display the pizza places in the area.

 

[SmokeTree]

This is how I handle this kind of thing with my own projects. Personally, I'd avoid parameters in the URL. Let's take your example site/com/find/rpgs/playstation. What you want is a "router" that handles HTTP GET requests and passes that on to a "controller" that will query the database and pass the result to your view/template layer.

Here's a short example of how I implement the router/controller part of a "pretty URL" system I have set up for music events. This code is in Elixir (http://elixir-lang.org/) with the Phoenix Framework (http://www.phoenixframework.org/), so it may or may not look familiar.

Say I want something like example.com/events/some-event-with-pretty-link . That's a lot better than doing example.com/events?id=42 Here's how I make it work . Starting with the router, I have this:

get "/events/:slug", EventController, :show, param: "slug"

What I'm doing there is I'm telling the router that URLs with a "slug" (think of it as a parameter) hanging off the end are to be fed to the "show" action of the Event Controller using the parameter name of "slug". Here's how things look with the controller.

def show(conn, %{"slug" => slug}) do
  import Ecto.Query
  event = Repo.get_by(Event, slug: slug)

  event = Repo.preload event, :venue
  conn
  |> assign(:title, event.name)
  |> assign(:description, event.meta_desc)
  |> assign(:canonical, event.slug)
  |> render("show.html", event: event)
end

What's going on there is I'm getting the event "slug" or the pretty link part (some-event-with-pretty-link) and I'm doing a database query for it. In the events table in my database, there is a slug field and in this case, using the example URL above, it would be holding "some-event-with-pretty-link". I'm also loading the associated venue with the event because that's how relational data works with Ecto (https://github.com/elixir-ecto/ecto). From there I'm sending the user's connection down a series of "pipes" to add the page title, description, canonical, etc until finally rendering the "show.html" template and also passing the "event" through the view.

This kind of design makes use of the REST architectural style (https://en.wikipedia.org/wiki/Representational_state_transfer). You can structure just about everything you are using parameters for this way. The only time I'll normally use parameters is for things like page number when I'm paginating records. Even though the example I gave use Elixir, the same concept works with many web frameworks. Hope this helps

 

[Ryuzaki]

SmokeTree's advice is next level. If you're prepared to dig that deep then I'm sure his method is best. He's the king.

I've done exactly what you're talking about with directories. My method is fairly simple. I did it with Wordpress but it can be done with any CMS. You can do posts, pages, custom post types, etc. Whatever keeps you organized, because there's a LOT of posts. If you can automate their creation that's even better.

So basically you could create a list of parent pages. 50 for the states. Then each can have published or draft posts for each location. You can pull them regardless. You can automate the creation with a CSV file and a plugin, which is probably best because you'll want to add meta data to each one using custom fields.

To go back to the original analogy, you might have custom fields for "console" and "genre", where "console" might be "state" which would assign it a parent page. Then "genre" could be "city" for the big city pages you want to make too. So when you mass-upload this content (the content can be in a CSV cell too, even including images with file names ad-libbed into the right spots with the right HTML tags surrounding them)...

When you do the upload all of the custom data fields are created and associated, and you can then create a custom loop in a custom page template that will create all of these pages for you. All it will ask of you is that you have the data uploaded. You can automatically filter with the custom data based on the State and City pages with their own custom fields feeding the loop the proper info, which then fetches all of the sub-posts like a blog loop.

It's critical to map out exactly what info you want in there first because there's no going back. So think about phone numbers, homepages, hours of operation, and all of that.

Then you can even go in and add a Google map to auto-populate locations based on auto-searching the address and feeding back coordinates. I had a Python bot coded for this I'll share with you if you want.

As far as the images go, like storefront pictures, sign pictures, logos.... you'll want to pay a handful of VA's to go find them. They WILL be lazy and miss some too, and you can then filter those out and have another VA go look to catch the ones they missed.

I then had another bot created that would create placeholder images of the ones that didn't have images. It would go Text-to-Image using the brand name. I will share this with you too if you want, it's Python.

It can be done on Wordpress relatively easy, it's just complex. You'll rank like gangbusters too, lots of long-tails and the main keywords will go down too with some interlinking.

 

[animalstyle]

SmokeTree thank you very much. I'm no developer, but I think I've managed to grasp the general idea.

I'm using Wordpress and it looks like I can use mod rewrite in my .htaccess file to accomplish a similar result.

Ryu, I am using custom post types with custom fields like you say. I'm using a plugin suite called WP Types that handles the custom post types, fields and outputting them. I've also done some of the same work with ACF and writing custom loops, but WP Types is easier for me.

I add a layer of caching on top of it with site wide preloading for site speed. With my limited technical knowledge, this is what I've found that works.

I'm going to dig into the automation methods you mention. Previously my databases have been small, but this project will be much much larger.

 

[Ryuzaki]

Heads up when you do dig, Wordpress allows you to create and add custom fields to posts without Advanced Custom Fields. You don't have to get too crazy on that part. You usually have to enable them as visible on the Wordpress Post Editor page using the menu at the very tip-top. It slides down and you can hide or show certain things. Custom fields are off by default.

 

[turbin3]

Sorry in advance if this isn't relevant. Just briefly glanced over the posts. Since you mention you're "not a developer", @animalstyle , another option that might work for you is WP All Import. Piece the data together in your CSV's, with your favorite spreadsheet program. You can categorize or tag things as you see fit, as well as setup WPAI to pretty much generate the structure of the site. On import, takes a minute to get your import template matched to the appropriate fields. Then WPAI will do the heavy lifting for you. Best part is, if you maintain the original import file, you can mod it, re-import, and run it again to update everything. Completely bypasses a lot of typical needs for dealing with databases, backend stuff, or programming logic. Obviously, there's a point of "mission creep" there, once you reach a certain point, but it can be excellent for rapid implementation versus endless coding.

For others that don't mind the investment of time and education, Ember.js has some awesome built-in functionality to do things exactly like this, pretty much exactly along the same lines as what SmokeTree mentioned. Here's an excellent example, especially considering the real-world application value he demonstrates (so helpful!), versus the theoretical esotericism so many often use as examples. Ember Data + external API's That'll get those "gears" turning on all that is possible. Just think of the applications!

 

[animalstyle]

Thanks @turbin3 I've been doing exactly what youve outlined. I use WP CSV (though going to check out WP Import All) and have been building a large file that I'll dump into the appropriate fields.

 

[Nat]

This thread is so awesome... thank you @turbin3 @SmokeTree & @Ryuzaki

@Ryuzaki now, about setting up that custom template/metadata/autofilling....

I've got a custom page/post template whatever.php. Right now I'm grabbing each metadata value under the header

/*
 * Template Name: Whatever
*/
<?php get_header(); ?>
<?php
$state = get_post_meta( get_the_ID(), 'the-state', true );
$city = get_post_meta( get_the_ID(), 'the-city', true );
...
$customer_quote = get_post_meta( get_the_ID(), 'cus-quote', true );

And then inserting the custom values into different spots within the template like

<div class="full-width" style="background-image: url('<?php echo $image; ?>');"></div>

<h2>Top Videogame Wholesale in <?php echo $city . $state;?> </h2>
       <p>Out of the <?php echo $num_stores;?> stores in <?php echo $state;?>, we believe that <?php echo $store;?> offers the best...[...]..

Do you leave <?php the_content(); ?> at the bottom of the page template for updating the post with new/additional information? Or is there an easy way to insert the info I was injecting into the template file into the content section? (does it even matter).

I've never done this before, but I assume there is an easy way to write another custom page template that acts like a category page for a specific post meta value? So, there could be a page that searches for every page with a specific $value for get_post_meta( get_the_ID(), 'some-key', true );?

Edit: Also, @Ryuzaki, when you said you ranked well by generate huge amounts of pages with semi-scraped/scraped data, how much content did each page end up having? Was it just short fields that were uniquely combined, or did you have unique 300 word descriptions?

Edit2: I also installed WP CSV and think I'm going to go with it... I haven't tried to generate a huge import yet, but it seems simple after doing an export.

Edit3: I'm going to do this with pages, not posts because I want to have a nested url structure and I don't want it interfering with the blog. Is there an easy way to get the wp_ID of a parent page? For example, I just realized if I generate 50 state pages, I can't just tag child pages as 'state' because the need a wp_ID parent.

 

[Ryuzaki]

Yes, what you're doing with the custom meta fields is fine. You can populate them automatically from the CSV. I don't remember exactly which plugin I used to import mine since I did it once and then exported the XML data and saved my wp-content folder.

But you should be able to import the sub-content (which comes together to form the child pages) as child pages of the child.... I'm not sure how you're building it but imagine this...

• State Page
  • City Page
    • 20 locations imported as separate pages, saved as drafts for organization purposes.

You might do this separately for every city, or if the plugin can handle it, direct the locations to be saved under the city page parent and upload them all at once.

I don't recall using the_content(). I autogenerated it all and my "content" was ad-libbed statements like you're talking about, right in the template.

Yes, you can write loops to pull posts based on custom fields. You can write one that says "only loop through this city's child pages that are drafts" to populate the locations. You can do all kinds of stuff using the same locations if you set up the meta data before the upload. Of course you can change and reimport if it doesn't get too sophisticated.

Yeah you can get the page_ID of the parent dynamically and have it pull all the locations. So every city and all of it's locations can be built off of one template.

global $post;
$direct_parent = $post->post_parent;

You request all of the data associated with the current post/page and then pluck out the parent. I'm not clear why you want that but you can. You can also definitely pull this off without the parent ID and using custom fields as state: XX. You can write loops based on custom fields all day long. Or upload as a a child of the parent. Or upload in pieces and associate them as you go.

Another option I wish I'd have done and will do if I ever do this again (I won't) is to upload all this as custom post types so it's not a giant mess in my posts or pages lists.

The amount of "content" on each page was like 100-150 words of adlibs, plus other types of enhanced content (not giving away all my secrets but I did allude to some on the forum at some point). Plus I had tons of 100% unique images and formatted data.