Step 3 – What we test
This is where we define the scope of the report. We need to establish what pages should be included in this site report, to ensure that the crawler recognizes it shouldn’t run off and try to test a whole social media site just because you linked to your company profile page on Instagram.
Whatever you set as the original homepage when adding the site will be carried through to this step automatically, defined as the home page in the forced pages, and a rule that allows for any URL that starts with that URL will be in the allowed rules section.
Allowed & Denied rules
Allowed rules define what is considered part of the website. So if you add an include rule of URL starting with https://example.com, it would include these pages in your report:
https://example.com/ https://example.com/archive https://example.com/blog/a-cool-post
Alternatively, you can exclude pages using the Denied rules. If you add an exclude rule of URL starting with https://example.com/archive it would ensure that these pages were not included in the report.
https://example.com/archive https://example.com/archive/another https://example.com/archive/and/another/example
This is quite comprehensive, the dropdown for the rules has URL starting with, URL ending with, URL contains, URL longer than/shorter than, and if you’re into that kind of thing, regular expressions.
This is an area where you can define certain pages (like the homepage, or a specific URL to use to test your 404 page), as well as add in pages that would not be found otherwise. If a page isn’t linked to in the content of your website, it is considered an ‘orphan’ page as it does not have a parent page linking to it. A simple example would be a landing page the marketing team set up for a PCC campaign. You wouldn’t be linking to it on the site, so it wouldn’t be found on a typical crawl of the site, but you may want to ensure the page is meeting your brand standards, so you could use the “normal” option in the dropdown and add those URLs into the report.