Xenu Link Sleuth to the rescue.
Xenu Link Sleuth is my favorite website problem checker. It is the best broken link checker I have found. I use it like a website problem checker. I can use it to answer questions like, "do I have any links that go to websites that are no longer online?"
 |
| Xenu Link Sleuth from home.snafu.de |
 |
| Link Juice of a different kind...(Photo by J. Kelly Brito on Unsplash) |
Do you know where you are going to?
Is your "link juice" flowing to places to which you had no idea you were linking? Let me explain. Website owners are frequently linking to places outside of their direct control.
The easiest way I found to avoid issues is to use Xenu Link Sleuth periodically, to run a report on the website.
It is multi-threaded, fast, and free. However, it only runs on Windows. You'll need to look for another tool if you need an online tool to extract links from a web page.
I used it a few hours to go to get a list of all links from my website. I had recently purchased a commercial theme for this website and after it was installed I had graphics and menus that I knew I had not uploaded showing up on some of my pages. So, I turned to link sleuth to make quick work of it.
The commercial theme was small and could not have uploaded images to my server. So, it was time to do some detective work.
 |
| Who are you linking to? Photo by Craig Whitehead on Unsplash |
I had no menu so my site was using theirs.
I discovered that there were many options linked to their hosts because I had not yet defined those options. For example, I had no "documentation" choice on my drop-down menu, but after installing their theme I had a new menu complete with irrelevant links, presumably as a demonstration?
I used Xenu Link Sleuth on my website.
Xenu Link Sleuth scanned my website producing a list of all URLs, including css, js, jpg, png, gif, etc. I was not pleased. I had to delete these links, but I had to find them. If one clicked on these links, one would find themselves looking a website theme store. Certainly, that was not the user experience I had intended to deliver.
By pasting the output into a spreadsheet, one can do some pretty elaborate filters, like show me all .jpg links being loaded from a particular domain.
Other uses for Xenu are to find orphan pages in a website. These are pages that don't aren't link to other pages on your site.
It is also a handy tool to get all links from a website. You can use it on websites with which you have no connection as a competitive intelligence tool.
Denial of Service Attack?
 |
| Tools can be misused...(Photo by Matt Seymour on Unsplash) |
Follow Robots.txt (or don't.)
Link sleuth, by default, will honor the robots.txt file, a file that webmasters configure to direct spiders to index only certain areas of the server. However, Link Sleuth can be configured to ignore robots.txt.
Put Xenu Link Sleuth in your toolbox.
I think you will find that Xenu Link Sleuth is a great addition to your website tools. I know this reads like an affiliate link post, but Xenu is not for sale, it is free.
