Have you ever googled your company name and come up with a result like this?
During a very busy summer, I had re-pointed my new domain name to my hosting account and when I entered http://jennsweb.net in my browser’s address bar, the result above was not what I wanted to see at all. Where did I go wrong?
Even though I have been doing this for a while, there are still terms like “robots.txt” and “.htaccess” that make me want to huddle up in a pillow fort for the rest of the day. Mainly I hesitate before dealing with things like this because involves mucking around with things like “root folder permissions” and “shell commands” and these are not exactly in my day-to-day wheelhouse.
So, I clicked on the words “Learn More” in the search result. That brought me to this page:
How I Fixed This
As much as we all depend on Google to guide us to what is safe and informative on the web, sometimes you can easily fall into a “Google Hole”. You start with one page in Google’s many, many subdomains and have to click on another page, and then another and before you know it, you are in a maze of browser tabs and links so deep you forgot what you even clicked on to begin your journey.
I decided to check in with my hosting company to see if there was some (hopefully) obvious reason my site was missing its robots.txt file. “But wait,” I thought, “What if this is somehow tied to the domain name re-pointing? “If I switch my domain name in my site’s General Settings to jennsweb.net won’t that affect my previous links? And hey, what about adding an SSL certificate to my site while I’m at it?”
I wrote down everything I wanted to try and figure out and opened up a chat with WPEngine. Because their customer support is amazing,I immediately got in touch with a technician named Clay. I wasn’t even sure which question came first: “What should I do first?? How do I upload a robots.txt file?? How the heck do I even create a robots.txt file???”
I found out that I had actually done things in the correct order, but just needed a few details done here and there. Here’s what I learned:
Fixing your domain url after re-pointing a new domain name
1: Get an SSL certificate installed on your site.
Some hosting companies charge an extra fee for this. Luckily, my hosting company will provide one as part of the hosting package. It’s worth doing. Here’s why. I wanted to get the proper syntax on the domain name before I made it official.
It’s a good idea to wait until your SSL certificate has been completely installed by your hosting company before going to Step 2.
2: Change your WordPress Address and your Site Address in your dashboard’s General Settings.
This is where your WordPress site decides what to call itself. I needed to change it in this case from “http://jennmearswebde.wpengine.com”, the default url issued by my hosting provider, to “https://jennsweb.net”. What’s the difference between your WordPress Address and your Site Address you ask? Basically, the WordPress address is for the dashboard of your site and the Site Address setting is for the “front-end” or, what your site’s visitors see.
Here is the updated setting for the site’s WordPress and Site addresses
TIP: Make sure you have your WordPress admin username and password stored somewhere handy. Once you change this setting, you will need to log back into your site due to the name being changed.
3: NOW, you can upload your new robots.txt file.
The reason I embarked on this journey in the first place! One thing I have learned in dealing with Google, it really helps to have all your ducks in a row BEFORE you submit anything to its Search Console tool. Otherwise, you try, it fails, and the great and powerful Google steamrolls on, leaving your site in the dust until it rolls around again.
How do you create a robot.txt file? What the heck IS a robots.txt file??
Googlebots and other types of webcrawlers, otherwise known as search engine spiders, are much like actual living spiders. They don’t really think too much. They simply sense movement nearby, go towards it, and try to sense if it’s prey or not. If it is, they pounce and feed. If not, they simply ignore it. Your robots.txt file is supposed to act as an instant indicator if your website is “prey” i.e. available to be indexed, (or not). So, you need to create a text file that contains some code that will let the webcrawler know which parts of your site you would like to show up on a Google search.
Before you create the file, make sure you can upload it to the root folder of your site. Your root folder is the topmost folder available to a browser. A lot of web hosting companies designate all your files to a root folder labeled “public_html”. If you aren’t sure which is your root folder, you can ask your hosting company.
WPBeginner has a very helpful article on what you need to include in your robots.txt file. You definitely want your site’s pages and posts and images to be searchable, but probably don’t want your plugin and theme files to show up on a SERP, so this is where you want to tell search engines exactly what is up for grabs.
For example, my robots.txt file ended up looking like this
4: Test and Submit your robots.txt file in Google.
To make sure that your new robots.txt file is functioning properly, Google offers this tool to test out your file. When you click on “Open Robots.txt tester” it will open a new tab that looks like this.
Confession time: my first attempt at creating a text file contained some hidden formatting code and my initial submission resulted in a list of errors. Google’s tester tool is pretty helpful in that it allows you to make the edits directly on the results page and then download the corrected file to then re-upload and test again. Here’s some guidance on creating a truly plain text file.
If you have more questions, check Google’s robots.txt faqs here.
So, what I thought was going to be a quick little file upload turned into a lot more. Sometimes a trip down a “Google Hole” is well worth the extra time.