Scraping from a Wordpress custom plugin

[bernard]

I've been writing my scrape code in Python (mostly Scrapy), which is cool, but I feel like its becoming sort of a pain, having it off site, with stuff like IP adress issues, having to FTP the data to my server etc.

I also have some other ideas for integration with Wordpress for a product, sort of like Google Site Kit.

My question is:

Is there any reason not to build a plugin in Wordpress that scrapes external data and stores it in a separate Wordpress table?

 

[Ryuzaki]

The first thing that comes to mind for me is resource demand and what this would do to your site speed and user experience.

The second thing would be burning your IP address in some fashion and ending up on browser's "be careful visiting this site" list.

 

[bernard]

The first thing that comes to mind for me is resource demand and what this would do to your site speed and user experience.

The second thing would be burning your IP address in some fashion and ending up on browser's "be careful visiting this site" list.

That's a very good point.

No easy shortcuts eh.

 

[adjacentbrowse]

Couldn't you just create your own API to access your scraped data when you need it? Or have the data in a Firebase database that you can access in realtime?