DubBot allows users to crawl websites via a specific domain and subsections off of a domain. In addition to these options, DubBot will now allow users to specify lists of URLs for crawling explicitly. This list may be provided via a URL to a sitemap, a CSV upload, or manual entry through the DubBot interface.
These new options for crawling content will allow users to curate content for auditing more selectively. DubBot users can use these options to help further prioritize content fixes.
DubBot users can find more information on how to configure these new options in the Help Center or reach out to our support team.
