I need to capture an entire website. How do I get all of the URLs?
If a user wishes to identify and capture an entire website, you will need to generate a list of each individual URL on the website. Numerous tools exist to generate a list of URLs on a website; two common options are listed below. This short list is not exhaustive—many other sites/software exist, and can be found by searching on “sitemap generator” or “web site crawler” in Google.
For both of the programs we list here, if you submit the main domain URL—for example, submitting the domain “xyz.com”—a list of all pages within “example.com” will be generated. (i.e. “example.com/about”, “example.com/contactus”, “example.com/products”, and the other pages within the “example.com” website.)
Below are two examples of resources that can generate a list of URLs based on a site (domain) name.
- Site crawler, requires a software download: https://www.screamingfrog.co.uk
- Requires a software download, maximum of 500 URLS in free version
- Site crawler, run through a browser: https://www.xml-sitemaps.com/
- Maximum of 500 URLs in free version
- After completing a crawl, scroll down to the “Other Downloads” section within the results page and select “Download All Sitemaps in a Zip File”. The enclosed “urllist.txt” file contains the list of crawled URLs.
How do I find the direct URL to a video or audio file?
If you want to download a video from an unsupported service, you will need to find the video’s direct source URL. Every website is different in how it is coded so these methods will not work for all websites.
Right-click on the video: Some video players are encoded with an option to “copy source URL.” If you right click on the video and this is enabled, click “copy source URL” and paste into Batch for the video.
Use the Video Download Helper plugin: Video Download Helper is a Chrome plugin that finds the source URL for a video. Please see the plugin’s documentation for the latest instructions on how to use it to find source URLs.
Inspect element: In Google Chrome, you can right-click on a video (or near it) and click “inspect element.” This will bring up the source code for the page and should highlight the code for what you selected with the right-click. You may need to expand further nested code to find a source tag (it will say “src=” in front of a URL), if it’s available.