how to find all pages on a website

Why Do You Need to Find Every Single Page on a Website?

Knowing all pages on a website can help you in several ways to prevent any ranking issues. It also helps you to see all the pages that search engines are seeing on a website in order to determine if any pages should not be ranked.

Based on this you can easily remove any unneeded pages to optimize your crawl budget and even improve your overall ranking by removing pages buried deep in SERP that are not supposed to be ranking.

However, it also helps you to do any SEO competitor analysis to see what pages your competitors have on their website, what SEO keywords they are trying to target, what pages are the most important for them and so much more.

So, let’s have a look at these 6 reasons why you should know how to find all pages on a website whether it is your website or not:

Reason #1: Find Orphan Pages

The first reason is to find any orphan pages on a website.

Orphan pages are pages without any internal links pointing to them from any other web page or section of a website, which makes the page inaccessible for website users by simply clicking through your website and even for crawlers as well.

how to find all pages on a website orphan pages

Usually, these pages are rarely to be indexed by search engines because they can’t be found by search engine crawlers as the prior way to discover web pages on a website is by crawling links.

to find all pages on a website search engines crawl links

However, even if they are indexed within the SERP, their ranking will often be very poor, and they can be lowering your average ranking position and even damage your overall ranking.

Thus, you want to know about these pages in order to make a decision whether to create internal links to the page, delete or remove them from the index.

Of course, not every page is meant to be indexed and often purposely made orphan pages such as landing pages for Google adverting or Facebook advertising are meant to be inaccessible from within the website.

Reason #2: Find Weird Ranking Pages

Another reason why you want to find all pages on a website is to find those weird, low-quality, thin-content, and spammy types of content that should not be indexed or better not exist on the website unless there is a good reason for them.

And there can be a lot of types of pages that should not be indexed and perhaps the best way to see and learn how to find all those weird pages is by looking at Glen from Detailed site audit videos like this one:

But for example, when you look at Backlinko, you can see one of their admin pages for a blog post was indexed by Google.

how to find all pages on a website weird ranking pages

And when I visit this page, it does not exist.

not existing indexed page for how to find all pages on a website

Of course, these kinds of web pages you don’t want to be indexed on Google, thus, should be removed as pages like this can waste server resources, can cause a significant delay in discovering important pages on a site, and plummet your ranking.

Reason #3: Removing Duplicated/Redundant Pages

Another reason why you want to know how to find all pages on a website is to remove any duplicated/redundant pages.

Duplicated content can cause a lot of problems for search engines and webmasters as well and if you want to ensure that you have applied the best SEO practices, you deal with them.

Now, when it comes to duplicated content, there can be two versions of that:

  • Within your website: This is when you have two identical pages on your website. This often confuses search engines on which one to rank, plus you might be wasting your crawl budget.
how to find all pages on a website to prevent duplicated content
  • Outside your website: This is when different websites copy content from your website and publish it on their own website. This can be a huge problem as they can over-rank your original content and steal your traffic. Even worst you might get a manual penalty for duplicating content.
example of duplicated content outside of your website

However, in this case, we are talking about duplicated pages within your website that you want to discover and remove, redirect or provide a canonical tag, so search engines know which one is the main version.

Duplicated content can be a real issue for your website and by knowing how to find all pages on a website you can fix issues like this.

Reason #4: Changing Website Architecture

The next reason why you would need to know how to find all the pages on a website is due to changing website architecture.

Website architecture for SEO is extremely important as it allows you to plan out all the different contents located on the website in the most efficient manner so that users can easily find the information they are looking for and search engines can properly discover, crawl, and index all your pages.

using how to find all pages on a website to optimize website architecture

However, one of the common issues that webmasters are facing later in the stage when their website has grown is poor website architecture and their SEO URLs structure that needs to be updated.

As most website owners do not really plan what keywords they will target, what SEO content they will have on the website, and other important aspects that come with designing your website architecture.

Thus, if you are in need of redesigning your website architecture, then you will need to know all the pages on your website in order to move your content around without recreating content that you have already on your website and so avoid keyword cannibalization and wasting your time.

Reason #5: Optimizing for Internal Linking Strategy

Another important reason why you would want to know how to find all pages on a website is when you want to optimize for an internal linking strategy.

Internal links are arguably one of the best SEO techniques to improve your impressions, ranking, and traffic to your website that you have absolute control over.

As mentioned, links are the prior way used by search engines to discover and crawl the website.

And internal links also pass link juice. Thus, with internal links, you can redistribute the link juice, AKA PageRank to other pages such as your product pages to boost their rankings.

how to find all pages on a website to improve internal linking strategy to pass link juice

So, with a properly optimized website with an internal linking strategy, you can easily increase your overall website ranking.

You can also help search engines to better understand your content when it is indexing it. Search engines are reading the internal links or the link text, which is called anchor text to get better context about the page.

google uses anchor text to better understand content on a website

Therefore, knowing all your pages and what keywords they are targeting you can essentially use that to interconnect the pages together when mentioning the keyword or topic within other pages like in this example.

using internal links with anchor texts for search engines to read it

Honestly, sometimes I play around with the text to mention the keyword on other pages. However, always try to keep it natural and within the context.

Reason #6: Doing Competitor Research

And the last reason why you should know how to find all pages on a website is when you want to do any competitor research such as SEO competitor analysis or backlink competitor analysis.

Your competitors are arguably one of the best sources of ideas and what works and what doesn’t for you to improve your SEO strategy.

And knowing how to find all pages on a competitor’s website will help you to find what pages are important for your competitors, what keywords bring them the most traffic and so much more.

Thus, knowing how to find all pages on a website will help you when you are doing competitor research as well.

So, now that we have discussed why you need to know how to find all pages on a website, let’s have a look at the different ways how you can do that:

Way #1: Using Competitive Research Tool

The first way to find all your and your competitor’s pages is by using a competitive research tool.

For this and other ways, I will be using SE Ranking and if you do not own any SEO tool or even if you have one, then I can definitely recommend you try SE Ranking as they offer over 30 SEO tools with the best value for money.

So let me show you how you can find all pages on a website using SE Ranking Competitive Research tool.

Step #1: Go into the Competitive Research tool, where you enter your or your competitor’s domain, select your target location if applicable and hit “analyze”.

using competitive research tool to find all pages on a website

Step #2: Here you will find a high-level overview of the analyzed website such as the domain trust, organic traffic, ranking keywords, the backlink profile, and more.

competitive research tool overview for how to find all pages on a website

So now, you just go to the “Top pages in organic search” report using one of these methods.

going to the top pages in organic search report in se ranking to find all the pages

And voila! Here you have the website pages of the analyzed website. Not all, but most of them. At least the most important for organic search.

You see, super easy. So, let’s move to the second method.

Way #2: Using Website Crawler

The second way to you can use to find all pages on a website is by using website crawlers and website audit tools.

Website crawlers usually work behind the scenes for SEO tools to discover website pages and backlinks and save them in their databases.

And website audit tools such as SE Ranking Website Audit tool are perfect for you to find all pages on a website as they work as a website crawler, plus show you all the SEO issues for every web page. And they are super easy to use as well.

website audit tool se ranking for how to find pages on a website

Step #1: To start using SE Ranking Website Audit tool, you just need to create a new project. This can be your website or any other website.

se ranking creating new project of find all the pages on a website

Step #2: Then you can “restart audit” most likely if you have created a new project then it will start automatically.

using website audit tool to find website pages

Step #3: Once the report is completed, then you go to the “Crawled Pages” report where you can find all pages on a website.

crawled pages using website audit tool for how to find pages on a website

Plus, you can easily customize the table with additional columns to get more information about the pages.

customizing website audit columns in se ranking for all pages on a website

This is a super helpful report for both fixing your web pages, learning more about the website’s overall architecture, internal linking strategy, and a great way to find all pages on a website.

Way #3: Using Sitemap & Sitemap Generator

The next way how to find all pages on a website is by looking at a sitemap or using a sitemap generator to create a sitemap for a website.

Most websites are having sitemap/s to help search engines discover new and newly updated pages on a website. Thus, improving the crawl efficiency.

using sitemap for how to find pages on a website

So, to find a sitemap on a website you can use a tool like Sitemap Test that helps you to analyze the website and show you any sitemaps located there.

sitemap test tool to find sitemap for how to find all pages on a website

Once you have the sitemap, then you can go through it and you will find all the indexable pages on a website.

how to find all pages on a website using sitemap

However, if there is no sitemap on the website, you can use a sitemap generator. For example, the SE Ranking Website Audit tool provides such a feature.

generate sitemap feature in se ranking for to find pages on a website

Or you can use free sitemap generators.

sitemap generator for how to find all pages

And if you are using CMS like WordPress you can easily install SEO plugins such as Rank Math that help you to create a sitemap for all your pages as well as create an image sitemap to improve the ranking of your images.

sitemap settings in rank math for how to find all website pages

Way #4: Using Webmaster Tools

The fourth way how to find all pages on a website is by using webmaster tools like Google Search Console or Bing Webmaster Tools.

Webmaster tools are made for website owners to see their website performance in the search results as well as any issues discovered by them or show notification of any manual action taken against your website.

Of course, if the webmaster console is owned by Google, then you will see only the performance for the Google search and likewise for Bing and other webmaster tools.

Now, because most of the webmaster tools are very similar to each other and because most of your traffic will probably come from Google as they own approximately 90% of the market, I will show you how to find all your website pages using Google Search Console.

google owns almost 90 percent of market share

So, when you open up Google Search Console, you can see the “Overview” report of your website. From here you will want to go to the “Coverage” report.

google search console overview report for how to find all webpages

In the coverage report, you will want to select the “Valid” pages and click on one of the options in the “Details” tab. Both reports together should give you all your website pages.

coverage report GSC for how to find website pages

If not, you can also select the “Excluded” tab to get more options to go through.

including excluded pages report in GSC to see all the website pages

Sometimes freshly new web pages can take a while before they will be indexed, or they can be excluded for reasons such as violating guidelines, adding a noindex tag, etc.

To learn more, about how Google finds all your pages check out this.

Way #5: Using Google Analytics

The next way to find all pages on a website is by using Google Analytics.

Google Analytics is great for finding orphan pages that do not have any links coming to them or are not included within the sitemap, and thus cannot be reachable by the crawlers.

Because, Google Analytics can access orphan pages if they receive at least one pageview, this can also include pageview from bots that are pretending to be the user.

With that, because there are two versions of Google Analytics, I’ll show you how you can find all pages on a website using each of them.

UA

To find all pages on your website using Google Universal Analytics go to Behavior → Site Content → All Pages.

using google UA for how to find all pages on a website

Now you want to change the date of the report from before you have launched your website and start using Google Analytics to today.

changing date in google analytics to find all website pages

Now, unfortunately, this is when you will also get a lot of trash that you will have to manually sort out from what’s the actual content page and what’s a URL of a site builder, search results, and other non-relevant URLs like in my case.

all pages on a website using google analytics universal

G4

Now let’s have a lot at how you can find all your pages on a website using Google Analytics 4.

To find all pages using G4, go to the Reports → Engagement → Pages and screens.

finding all pages on a website using Google Analytics 4

Then you will want to change the settings to see page paths or the URL of your web pages if you don’t the page title of the same page can change based on the translation of the visitor.

changing to page path setting in g4 for to find all pages

You will want to change the setting from “Page title and screen class” to “Page path + query string and screen class”.

g4 analytics setting to page path and query string for all pages on a website

And lastly, you will want to change the date of the report from before you launch your website to today just like in the previous example.

changing report date range for to find all pages on a website using G4

And again, you will need to manually sort through the report to find actual content on the website and not search result pages, page builders, and other not relevant URLs.

how to find all pages on a website using google analytics 4

Way #6: Using Site Operator

The next way to find pages on a website is by using the google site operator.

Simply, just go to the Google search and write site:yourdomain.com for example in my case it would be site:b2bdigitalmarketers.com

using google search operator for how to find all pages on a website

As you can see, all the search results are only from the specified domain.

site operator search results to find all pages on a website

And if I would remove the site operator you can see Google gives the results that can include other websites and not just mine.

without the search operator google provides other website pages as well

Now I want to give you a tip on how to find quickly internal linking opportunities whenever you publish new content on your website.

For example, recently I published a new article on Blogger Outreach, so to find quickly any keywords mentioned within my other articles I can use this operator and add the keyword behind it.

finding all pages on a website that mention a specific keyword

And just like this, I can find all the pages on a website that mention any specific keyword and quickly create relevant internal links for every of my new web page.

finding all pages on a website with a specific keyword mentioned within

Way #7: Using Robots.txt

Another way to find all pages on a website is by using the robots.txt file.

Robots.txt file is used by website owners to manage access for crawlers which sites, directories, subdomains, and categories can and cannot be accessed.

using robots file to manage access to your website pages

Therefore, by using the robots.txt file you can find any hidden pages on a website from search engines and crawlers.

So, to find the robot.txt file, type into the search bar: example.com/robots.txt for example in my case it would be b2bdigitalmarketers.com/robots.txt.

robot.txt file to find all hidden pages on a website

As you can see, the only disallow directory on my website is the WordPress admin section.

Way #8: Using CMS

The next way to find all website pages is by using your own CMS (content management system) that you run on your website.

Now, when it comes to CMS, there are several different versions you might be using such as WordPress, which is the most popular, and then others like Joomla, Drupal, or HubSpot CMS.

Each of these are slightly different but should give you the same information about your website pages. However, for this tutorial, I’ll show you only how to find all pages on a website using WordPress CMS.

Step #1: Login into your WP dashboard.

login into dashboard to find all pages on a website

Step #2: In your sidebar, you will find a couple of options where you can find different website pages:

  • Posts: Here you can find all your blog posts pages.
  • Pages: Here you can find all website pages that are not blog posts.
  • Portfolio: Here you can find your web pages for your different portfolios like pictures and videos, and some are even used for customers or to showcase any of your projects.
  • Others: Then depending on your website purpose there can be other options like forums pages, eCommerce product pages, customer ticket portals, etc.
how to find all pages on a website using wordpress cms

All these options should have all your pages. You can also look at the “At a Glance” box that should have all your pages and options on where to find all your pages.

It’s also important to look at the different sections of each of these options. For example, the Posts option also has:

  • All Posts: This is where you find all your blog posts.
  • Categories: This is where you create categories to group your blog posts. Each category has its page.
  • Tags: Each blog post can have assigned different tags that work like topics that could be associated with relevant content. Something like #hashtags. Again, each of these tags are having a separate web page.
wordpress post options with different sections for to find different web pages on your website

Quick Tip: Usually, I put noindex for tags as they have very little value for users, thus they would just hurt my overall ranking.

You can also use a WordPress plugin such as Show Pages URL List that allows you to view all the pages on your WordPress site and view the URLs as well.

Way #9: Using Other Tools

And the last way how to find all pages on a website is by using other tools, crawlers, scrapers, etc.

There are perhaps thousands of different tools that can scan your website to find all your pages and I obviously won’t mention every single one of them.

However, there are two more tools that I still want to mention, and that are:

  • Screaming frog: This is probably the most famous and best spider website crawler on the market for finding any issues on your website as they provide an extensive amount of information about the crawled website. The tool is especially useful once your website crosses over 1,000 pages.
screamingfrog for how to find all pages on a website
  • Netpeak Spider: This is another famous and excellent website crawler that helps you side audit your website and of course discover all pages on a website. Their feature-packed tool not only helps you to find all your pages but also optimizes them for SEO.
netpeak for how to find all pages on a website

And then there are other ways and more technical that I did not mention and perhaps even don’t know of.

However, I believe these nine ways should be enough to find all pages on a website.

Final Advice

Now, most likely you will need to use several of these methods to find all pages on a website, and depending on the size of your website you might need to go for the more technical ways.

However, I am confident you should know how to find all website pages on any website.

Did I forget something, did you learn something new, or do you have some comments? Please leave them on my YouTube Channel, CAU!

Related Articles:

Also, check out our SEO hub page to find all our SEO Resources.

Disclaimer

Some of my links are affiliate links, which means if you purchase something, I might get some small commission as a reward for reference. Of course, I am actively using all these services and products, and I only affiliate products or services I have full trust in their quality!