The Screaming Frog SEO Spider is a website crawler, that allows you to crawl websites’ URLs and fetch key onsite elements to analyse onsite SEO.
The SEO Spider is powerful, flexible and able to crawl both small and very large websites efficiently, allowing you to analyse the results in real-time. It gathers key onsite data to allow SEOs to make informed decisions. Some of the common uses include:
Find Broken Links.
Audit Redirects.
Analyse Page Titles & Meta Data.
Discover Duplicate Content.
Extract Data with XPath.
Review Robots & Directives.
Generate XML Sitemaps.
Integrate with Google Analytics.
Screaming Frog SEO Spider:
https://www.screamingfrog.co.uk/seo-spider
(Download for free, or purchase a licence for additional advanced features)