The Technology Behind Our Niche Sites
Running an army of niche sites can be quite a daunting task, even for those who are tech savvy. In this post, I’ll cover the basics of what you need to run a large number of sites effectively, plus throw in a few extras.
As many of you know we host out sites with GoDaddy on a shared hosting account using their Ultimate 4GH package. We actually have a couple of shared hosting accounts setup and spread our sites pretty evenly among them. We have found that 350-400 niche sites is really the maximum you want to put on a shared hosting account at any one time. GoDaddy recommends 200 and your results will vary depending on traffic and content. If you use video or a lot of graphics expect to to have less sites on each shared hosting account.
Administration becomes a bit cumbersome with so many sites in the same account, So I suggest using a useful naming description when setting things like databases or folders for your sites. Use the site names or a variation thereof if you cannot exactly match the domain name. Trust me, this comes in handy when trying to go through several hundred databases hunting for a particular site.
But why GoDaddy? There are cheaper solutions out there, but GoDaddy’s support is top notch. It’s 24/7 which is nice for someone who spends most his time on the other side of world, 12-16 hours ahead of the US depending on time zone and daylight savings. We have become a big enough customer that we have a dedicated representative to call with issues. No more hunting through phone menus or waiting on hold.
The GoDaddy cPanel has it quirks. I sometimes have to hunt through different screens to find the option I need and it can time out at the worst opportunity. However, the ability to add Account Executives and give them privileges based on needs is quite powerful for a shared hosting platform. I almost feel as if it was made with outsourcing in mind.
GoDaddy does have a one click install for WordPress that is good for a novice user or if you are only setting up a few sites. If you are setting up a large amount of sites frequently, I suggest using a custom install and doing it yourself. Setting up databases is a manual process that is tedious, but the rest can be automated.
I also like the fact that GoDaddy gives SSH access to your shared hosting account. For those of us more comfortable with command line access, installing WordPress and moving files around is a breeze. I even have a script and a custom WP install zip file that sets up all plugins needed and has our themes modified automatically. A big time saver. Admittedly other providers have this access, but it’s not as easy to turn on (or off) as it is with GoDaddy.
Lastly, we flip a lot of sites and most people have a GoDaddy account, so migration is easy. I am going to do another post on our migration process soon, but whenever different hosts and registrars get involved the error rate goes up leading to downtime and lost revenue. GoDaddy to GoDaddy site transfers are just more reliable.
Content Management System
All of our sites are built on WordPress. The number of plugins available and the ease of getting sites setup quickly is invaluable when you are setting up so many new sites every week like we do. Posting content is point and click so even non-technical staff members can do it.
We use a number of plugins on every site, but I have tried to limit their use to only the core requirements. Having too many plugins on your site can slow things down and unnecessarily bloat the file system. I have removed the default plugins (like Akismet) from our installation of WordPress because we don’t need them (we don’t allow comments on our niche sites).
Here a list of our standard plugins and why we use them:
- All in One SEO Pack — Great for adding meta data to pages and posts.
- Contact Form 7 — We use this for our contact us page that we put on every site.
- Date Exclusion SEO — Removes the date from posts making your site look fresher in the SERPs.
- Login LockDown — Critical to keep away brute force attempts to break your admin password.
- ManageWP Worker — Allows me to control all WP install through one centralized dashboard. See more below.
- Really Simple CAPTCHA — Adds CAPTCHA capability to our contract forms preventing us from being overrun by spam.
- WP Security Scan — Gives me a good idea if a missed anything security wise with my site.
- XML Sitemap Feed — Creates automated XML sitemaps that can be picked by the search engines or submitted manually.
One of readers, Steve, of Cloud Ventures Group, mentioned a new service called ManageWP in recent post. ManageWP allows you to consolidate WordPress dashboards all in one place. It’s still in beta and free to sign up, though you can only add a maximum of 100 sites at this time. This hosted solution is actually built around the ManageWP Enterprise Self-Hosted software
I have been “kicking the tires” pretty hard and ManageWP has pretty much stood up to every test. It uses a worker plugin to connect to each site. This saves you from having to enter your credentials into ManageWP and makes for a more secure connection overall. One note, however: be sure to add your site to your ManageWP dashboard after activating the worker plugin. This will close the loop and ensure no loose ends are left for someone to exploit.
The power of ManageWP is really in it’s ability to give you a quick snapshot of what’s going on with all your sites. You can see what sites need a WordPress update, plugin update or even theme update and then update them all at once. You can even connect to your Google Analytics account and have a stats snapshot on your dashboard. Backups, posts, pages, plugins, themes and users can be added in bulk, making it easy to make a change across all your sites.
Sub-users can be added so outsourced staff can have access to your sites, without having a username or password directly. This makes it more secure, because you can revoke access to all sites at once if needed. The sub-user section still needs some work, as they cannot do mass updates or use groups, but I am sure the functionality is on the way.
There are still some bugs, including importing sites and the aforementioned lack of subuser feature/functionality, but overall I feel this will be an extremely useful product for those of us with a large number of niche sites running WordPress.
I use a variety of tools to move or edit files and usually like to stick with free opensource software. Here are a few of my favorites:
- notepad++ — Great for editing config files or code snippets. Keeps everything well organized and highlights opening/closing tags.
- FileZilla — The de facto standard FTP client. Nothing is better for quickly and easily transferring files.
- 7-Zip — An extremely fast zip program with a nice interface making it easy to update compressed files.
- PuTTY — Simple, but full featured, SSH terminal software. Doesn’t require an install either.
For syncing files between agents’ computers or sending customers a copy of there purcased sites after a flip, we use DropBox which I’m sure most of you are familiar with or have at least heard of. The desktop app allows you to selective sync certain folders which is great for saving disk space. Let’s face it, you really do not need access to all those old files!
That covers a good portion of the technology we use with our niche sites. Is there anything you think we’re missing or would like to share with others that helps you?
Social Media is here to stay at least in some form or another. No, there isn’t something out on the horizon that I know about, it’s just that nothing is ever for sure or for certain.Thanks for sharing.
I like to hand-code my sites, and I’ve studied ways to improve overall site performance.
Let’s make the web faster
Best Practices for Speeding Up Your Web Site
Tools I use:
Notepad++ (text editor)
Komodo Edit (IDE)
WinSCP (FileZilla alternative)
Firebug (Firefox plugin)
Pagespeed (Firefox/Chrome plugin)
Opera browser (and Opera Dragonfly included)
Fireworks (old copy from 10 years ago, but still works 😉
Hey Guys, great blog. I’m getting a lot of tips here.
Can I ask about your domain buying? Do you use the discount domain club?
Whatt about whois privacy protection – that seems very expensive from Godaddy. Are you buying it, and if so how much does it cost you for each domain?
Yes, we use the discount club and get free privacy protection when we buy in bulk. On renewals too. Just one more reason I like GoDaddy.
As Joe said, the discount domain club is totally worth it if you’re buying domains at the level we are. Once you pass a certain threshold you get a dedicated Account Exec too which is nice. I think it’s 6 or more domains at the same time that allows you to get the free Whois privacy protection, but I’m not 100% on that…
thanks for the tip regarding the date exclusion seo plugin. I would have never thought about that otherwise. Take care. Kevin
You’re welcome Kevin. Always here to help!
I hope ManageWP Worker will be always free for 100 or less websites 🙂
Well they need to make money somehow and most people have less than 100 sites. So expect a minimal fee.
Hey Joe nice post. I have gone a bit nuts on your blog this week I have almost read every post you guys have put out and now i want to take some action. Btw i love how you guys are 100% no BS. So myself and John want to create some projects for Idle staff and I am thinking Niche site creation could be one option. However im sure we would not do it at the same level as you guys and might consider a little different approach. One I am considering is creating niche site’s then turning them into an e-commerce site if we could get high amounts of traffic. Or just going for more content and more value for people visiting the site’s rather then have 100,000 sites like you guys ;). I noticed its something you are beginning to do anyway right? correct me if im wrong.
Anyway the first step is just to start taking action and then go from there. right now im not sure what im doing and might only make 1 site and stop or could end up making 100+ not sure at this stage. We really want to add value for our staff by adding job security so we want to have some long term internal projects. This is the main reason for considering doing something like this. Also I was wondering if you still use godaddy.com to host? John was thinking you use bluehost.com or something else. But after reading this post i can see your reasons for using godaddy.com.
for any of your readers reading this comment just quick FYI… I am a partner in a small outsourcing company here in Davao City same place as Joe and Justin. I have been lucky enough to meet these guys here in Davao and can tell you they are such genuine guys and are real entrepreneurs. I have been living here 6months now and never want to return home unless its on my terms so i want to take any action possible to achieve that goal. This is one reason i am considering doing a new project like creating niche sites. Also got to make it to Brazil world cup in 2014 🙂
Matt thanks for stopping by!
We’re currently testing authority sites, but not exactly ecommerce. That’s probably something we’ll but an MBA in charge of next year.
We currently use GoDaddy, but I am thinking of testing a new provider soon. We use multiple hosting account so we have different IPs.
Found your site about a week back and have been reading your stuff non stop!
You said that you use the CTR Theme for most of your sites now, do you find a lot of issues with it? After I create a page and enable the Adsense option, the copy from the page does not display? If I disable ad serving the pages display fine. I have the same issue with both the sites I setup to test the theme. I am new to WP and was hopping this theme would make it a lot easier for me to build sites but it’s turning out to be rather frustrating :-/
Thanks again for all the valuable info!
Be sure to remove any scripting in the actual pages (including any adsense codes and Google Analytics) as that seem to interfere with CTR Theme. Also some times you need to play with the layout to get it to work right. The cool thing is once you get it right it should be easy to roll out those changes to all sites.
Thanks Joe! After I read your comment I went back and took a look at the adsense codes and I am pretty sure I put one of the codes into the wrong spot so the ad size was off and messing everything up… all fixed now!
Have you guys done enough testing to figure out which add layout performs best and settled on specific layout, or do you still let it rotate through all of the ad layout choices?
Thank you very much!
Ah, the layout test. We have been trying to scientifically test this, but I am coming to believe more and more there may be a big in how CTR Theme reports this stuff to AdSense using custom channels. Also, CTR is so dependent on advertisers and niches that unless you test each site with a different theme, you results will always be called into question.
That aside, over the last 90 days the best performing layout by CTR has been the 728 Top followed by the the 336 Right. They significantly beat all other layouts including the default Roadblock.
This seems like a good idea for a post. Thanks Kevin!
Hmm… 728 Top & 336 Right is currently working out for ya, good to know! Definitely would enjoy a detailed post on this topic.
Thanks for the info once again!
‘Blocking the privacy-policy page from getting indexed’ – Very good suggestion. I didn’t think about that.
Thank you very much
Joe or Justin
Is there any way of being notified replies to this post? I can’t seem to find where you click to be notified of replies.
WordPress should automatically update you when you post a comment or reply to one. If not I will have to look into the options.
Great post, good to see you’re sharing some of your secrets with the rest of the community.
I thought I was the only one that knew about ManageWP! 🙂
I like the idea of using Date Exclusion SEO plugin and I think I’ll give it a try, thanks.
Thanks Phil. They haven’t really promoted ManageWP so it’s not surprising so few people know about it. I just hope it doesn’t wind up costing too much when it comes out of beta.
great site and info you provide….curious do you use any site cloning like WP twin/backup buddy to quickly create new sites ? i also just saw a WSO for a great backup/cloning plugin for $17
We don’t. My staff and I do everything manually. It really only takes a bit more time once you have the process down, but is much more reliable. Too much reliance on automated tools can leave you high and dry when something goes wrong.
ManageWP has a cloning feature and is free!
Sounds to me like a theme issue. Try these steps:
What theme are you using? I highly suggest ProSense-Blue or CTR Theme, neither have this issue.
Does this happen to posts and pages? Does the date still show in the SERP? If yes to both questions, something is wrong, try reinstalling the plugin. If that doesn’t work you have a theme conflict.
Is this happening only on the actual post page or on the front page as well? If it’s only on the post page it’s not that big of a deal.
Are you logged in when checking the page? If so, log out and look at the page. Sometimes WordPress put the author and time stamp at the bottom of the post when you view it while logged in.
If none of this helps, I will need to take a closer look at the site. Do you mind sharing?
Had most of these installed on my niche sites, except for a few. Glad I came across this post.
Joe, I just installed the Date Exclusion plugin, but when activating it, at the top of my posts I still see “by on”. It would usually say “Posted by Admin on January 12th, 2011”.
Any idea how to remove the words “by on” from appearing at the top of the posts?
Sorry, can’t help you there…better question for Joe to answer…Joe, any thoughts?
This was a good read for me, it was interesting to see that you guys are using a lot of the same things, plugins, etc., that I use on my own sites. With the exception of the ManageWP, I had not heard of that one yet but on my way over to check it out now.
I also looked around the blog a bit but couldn’t find any reference to what you guys are using for link building to your niche sites. Any tips you could share on maybe the services, processes that you use to build links and rank your pages?
There has been a lot of interest in a link building post. It’s behind a few other things, but look for it soon!
First of all I want to say thank you for inspiring me to jump in this flippa business.
What I always wondering is how you can optimize and maintenance each site since you have hundreds site and each site have more than one article, if you have VA how many people handling those jobs?
Thanks for stopping by! I just noticed your niche site is at the top of the second page…congratulations, almost there!
We currently have 6-7 people working on the project, but a few of them actually order things from other outsourcers as needed. It can be a pain going back through the sites and updating information, plugins, etc. but I think it’s our tracking and organization that allows us to do it on a larger scale. Ordering content via email and trying to keep everything in a folder would be quite miserable and almost impossible with this many sites.
Amazingly, we’re still mostly organized via Google Docs, although we keep reminding ourselves we really need to get a CRM. We’ve kind of put this off because setting up a CRM just the way you need it is quite a project in itself and we simply haven’t had the time to sit down and run through it yet.
I would just reiterate what Justin said, by saying stay organized. Whether you use a CRM like vTiger, a project management system like Basecamp, or even just a Google Doc, staying organized and on top of your personnel is key. That’s why you are the boss and make the big bucks right? 😉
We often get outsourcing deals from people who don’t have a plan or a process. They’re completely unorganized and because of it, we don’t take their money. The project would fail and they would blame us. You can’t expect workers to have your overall vision or strategy. That you need to come up with yourself.
Thanks Joe and Justin for your respond of my comment.
@Justin I am really lucky because with just simple off site optimizing my site goes to page 2 and I hope by the end of September it will page one.
Google docs also help me to organize my work but since I have to many website so its really miserable, that is why I keep tracking everything with manual writing on whiteboard
@Joe As I remember that you have a team in Philippines, but I see lot of people using fiverr. For off site optimizing I still afraid outsouring is good choice because it can harm our site, do you have other opinion or tips for outsourcing off site optimizing?
Justin is working on an offsite link building post. Look for it soon.
In short though, I would say make sure you know the process yourself very well, then try to outsource it. Don’t just find people willy nilly on Fiverr and expect great results. That won’t happen. The other option is to hire well known services that have a proven track record. A lot more expensive, but can be worth it it you need links built right and don’t have the staff hired or process down yourself.
Thanks for the insight Joe. I am a huge fan of the site. I would love to get more details on how to do a custom install of WP. Currently, I use the SimpleScripts program within Bluehost’s CPanel. It’s probably similar to the simple install option in GoDaddy. I’ve poked around in MySQL and read some tutorials on how to install a new database and add the WP files to the database, but it’s still a little too intimidating for me.
I think I need to modify the WP files to include the database information before FTPing them into the proper file. I’m just not clear on what to change, exactly where to put it, what to name things, how to make it secure, etc. Also, it would be great to learn how to install my modified themes already configured with all the plugins configured. Couldn’t I just set up one site exactly the way I want it, then copy the Plugins and Themes folders from that site into the new WP install (essentially copying over the default files with the already configured ones)? I would love to have a script that does all that for me and actually know how to use it.
You touched on a few places that take a considerable amount of time for a newbie such as myself to figure out. Installing WP, configuring WP, customizing and configuring themes, installing and configuring plugins, constantly updating everything and knowing when to update, and backing up all your sites. It’s such a pain to update everything manually. ManageWP looks like a great solution to the update, backup, and easy cloning issues I’m having. My #1 hesitation is not knowing even a ballpark price before investing the time to set it up on all my sites. I’m afraid I’ll love it and then find out it’s out of my price range. #2 is the concern that while I can update all my sites as once, I could also screw them all up at once.
Anyway, thanks for all you’ve done. I’m kind of trying to get out of the rat race and build a better life for my family on my own; crappy job by day, building sites by the moonlight. You guys are like my unofficial business partners. I really appreciate it. Take care.
Chris, glad I could be of help.
A detailed WP installation post would probably be beneficial to the community here, so it’s something to put on the to do list (which is getting very long). Thanks for the idea! Perhaps I will include some screencasts to make the walk through easier.
You’re on the right track with your method. Get away from scripts your host might offer. While they may save time short term, the installation is so vanilla you still need to install the plugins and themes manually. You don’t need to modify the WP files before FTPing them to the site. The WP 5 minute install will create a proper wp-config.php. So all you need to do is create the database, download the latest zip file from WordPress.org, install the plugins and themes in the proper wp-include directory, and then send it over to the server. Then go to the site and you will get a WP error with a button provided to start the install process. Make sure you have your database particulars from Bluehost, like DB name, DB username, DB password, and DB hostname (this could be localhost). Viola, WP is setup and all your plugins/themes are available (though plugins still need to be activated)!
If you had SSH access and were familiar with a UNIX command line prompt, you could drop the zip file in place and have a script unzip it to the proper directories. You could even have the script modify the wp-config.php files — if the DB credentials were passed to it or known from a data file. This would basically give you a custom one step install script that gets WP setup your way!
Can somebody please tell me why I didn’t know about ManageWP before I got your newsletter today??? You learn something new every single day… will definitely be playing around with this tool – looks incredible.
The type of tool any niche blog marketer needs in their arsenal.
Props – Thanks for Sharing! 🙂
Thank you, for stopping by. Please let us know your experience with ManageWP and be sure to report any bug so the developers can make it a better tool for everyone.
I have been using ManageWP for a couple of months now and love it as well! I got mine upgraded to manage 200 sites. Great post, you use a few plugins that I don’t…I will look into these…thanks!
Thanks Spencer, I too got upgraded, but figured I might not want to let the word out and have the ManageWP team get inundated with requests. Now they need to go to 1000 sites so I can stop giving out direct access to my sites!
Really nice…we really like it. Hope they don’t charge an arm and a leg when it’s fully released…ugh.
I gotta agree with Halfdan..
This post only seems like half of the picture. I’d love to see an in-depth post on how you get your sites indexed and ranked, rather than just the WP install and plugins..
Understandable, and we will have a marketing focused post coming up. The point of this post, however, was to help those struggling with what tools to use to setup, manage, and maintain the sites.
Thanks for the mention on the Mangewp comment..
Its really great when the message gets out to people on such a great tool.
I can only hope they set the price at a reasonable level wehn its live. even with 40 sites its a huge time saver and easy to just pop in and do upgrades.
I have it set to email me when there are updates available. Thats a flaky part of the system at them moment as it doesnot always recognise all install need doing atthe same time for a plugin.
Steve, no thank you for bringing this valuable tool to my attention!
I encourage you to email the ManageWP team with any bugs or issues you might encounter. They are extremely good about getting back to you, but just might have overlooked some part of the testing. That’s why it’s still in beta!
I sure do 🙂
Have you noticed the happy and sad face at the bottom of the app.
Ive used it a few times and a sad face message gets a very fast response 🙂
A very nice touch in a great app!
Yes, I noticed that too. Great idea. They are on top of any feedback.
One thing, you “forgot” to mention – or decided was not appropriate for this post – is how you index your sites, once their bold. I recon reading, you used OnlyWire to bookmark them. Do you still do that – and isn’t it a bit pricery for such “simple” a task compared to services as Linklicious?
Another good suggestion, Halfdan, thanks. I really wanted this post to focus more on tools I use to manage, setup, and maintain the sites as opposed to marketing tools like Onlywire (which we still use). However, we have had some interest in our link building strategy, so look for a post soon. We are actually changing the process a bit right now, so we don’t want to release anything until we have it down.
I see – I’m looking forward to that post in that case. Thanks for releasing all the information, once again! 🙂
you might wanna research for a cache plugin for all your WP sites. you’re not updating those sites, well at least not that often, so why not save some server resources?
We have used WP Super cache in the past, but out sites are so lightweight I think it is detrimental to cache them at this point.
with filezilla, be careful, don’t save any passwords inside. those passwords are kept into a simple xml file with no protection/encryption at all.
A couple of years ago there was an infection running around, a virus that was copying the ftp information from that xml file then upload on those sites in every html/php/js file an injection code that infected further more the users/surfers computers…and so on.
It happened to me too and its not fun trying to clean hundreds of thousands of files from several hosts…especially if you have no clue like I was, without counting that if google sees those infected files will put your site on the infected sites pages.
Further more, I would also suggest to not keep inside the filezilla the ip of the ftp server. Edit your windows localfile for that, add the ip there and give it a name you would recognize…like
and then in filezilla, instead of putting the IP, put bluehost. Like that, even if your filezilla file gets hacked…well, you add another easy layer of protection
Good suggestions, there is always some level of risk when saving passwords into any software. I always yell at Justin when he saves them in his browser!
However, FileZilla has changed their storage method to encrypted and using the windows registry. Still if someone gets access to you computer, they would gain access to your FTP. So caution is always advised.
Do you have an insight on what ManageWP will cost in 3 months? Free now is great but when the time comes to charge I wonder what the cost will be.
Also great site, have been reading it for sometime now. I would like to see the spreadsheets you guys use to keep track of all the data that flows your way.
No insight as of yet, but have requested an unlimited pricing plan. I hopeful it will be affordable because it really is a great product.
I am doing an AdSense organization post now, look for it soon. As for our spreadsheet it’s fairly complicated though I might share the headers here soon. We are thinking of moving to a CRM soon however because it is just getting to unwieldy to tracking all this stuff in a Google Doc.
Thanks for sharing the manageWP.
It is an awesome management system. I will definitely use it because I always forget my passwords.
And you can do backup also.
No problem Lemuel, let us know you experience with ManageWP. If you have any comments feel free to share.
Wow, thanks for sharing ManageWP. I am so excited to come across that, going to save hours of work. Can’t wait to see the limit of 100 sites upped though once it’s out of beta.
Contact support and they may be able to raise your limit to 250 sites. Vladimir Prelovac personally answers emails and is very open to suggestions and bug reports. But yes, I can’t wait for the system to go live either!
Wow, thanks so much for letting me know about ManageWP! This will be a huge time saver for me since there seems to be a wordpress, theme, or plugin update every 5 minutes and I spent a lot of time logging in to each site and updating.
Just wondering if you guys routinely back up your sites or back up the databases at all? I currently don’t because my niche sites don’t have a lot of content. I was wondering if I should be doing this though.
You’re welcome Nicole, I was impressed when Steve recommended this product too. I am hopeful it will become our main way to administer sites.
We don’t currently do backups of our sites because we have copies of all content, including images, in a DropBox folder. So if we lost a site (which, knock on wood, has not happened to us so far), we could simply rebuild the entire site from scratch. I suggest you do the same!
I’e been wanting to ask you to make such a post for a long time, but it seems I wasn’t the only one wanting such information – thanks alot. Putting date exluciosn on my sites as we speak! 🙂
No problem Halfdan, please let us know if you have any other suggestions. Date exclusion is great — one of our staff members actually found it, too!
Wow! Didn’t knew about ManageWP, thanks for the notice, going to check it out right now.
Keep up the posts, I love them and they’re really inspiring!