Assistant answers: URL scanning

URL scanning allows you to automatically discover and index content from websites by simply providing a list of starting URLs.

For example, if you paste in https://theta.co.nz the system will:

Scan that page for content
Find all links on that page
Follow those links (if they're on the same website)
Continue this process to build a comprehensive knowledge base, which can then be used by your chatbot's AI assistant to help answer user questions.

How it works

Starting the scan

You provide a list of URLs to scan, one per line, for example:

The system will process each URL and automatically discover related content.

Automatic content discovery

When scanning https://theta.co.nz/products the system will:

Extract the page content - All text, headings, and useful information
Find all links - Every clickable link on the page
Follow relevant links - Only links that meet the inclusion rules (see below)
Repeat the process - Continue scanning newly discovered pages

What gets included vs excluded

✅ Pages that get scanned

Same domain pages If you start with https://theta.co.nz, these will be included:

Sub-path pages If you start with https://theta.co.nz/support, these will be included:

Document files

PDF files: https://theta.co.nz/manual.pdf
Word documents: https://theta.co.nz/guide.docx
Text files: https://theta.co.nz/changelog.txt

Processable extensions (downloaded and indexed):

Documents: .pdf, .doc, .docx, .txt
Web content: .html, .htm

❌ Pages that get ignored

Different domains

If you start with https://theta.co.nz, these will be ignored:

https://microsoft.com (completely different domain)
https://support.google.com (different domain)
https://dash.theta.co.nz (different subdomain)

Parent Paths

If you start with https://theta.co.nz/support/guides, these will be ignored:

https://theta.co.nz/about (above the starting path)
https://theta.co.nz/products (above the starting path)
https://theta.co.nz/manual.pdf (above the starting path)

Binary Files

Images: https://theta.co.nz/logo.png
Videos: https://theta.co.nz/demo.mp4
Software: https://theta.co.nz/installer.exe
Archives: https://theta.co.nz/backup.zip

Excluded Extensions (skipped during crawling):

Images: .png, .jpg, .jpeg, .gif, .bmp, .tiff, .ico, .svg
Spreadsheets: .xls, .xlsx
Archives: .zip, .rar, .gz, .tar, .7z
Media: .mp3, .mp4, .avi, .mov, .flv, .wmv
Executables: .exe, .dll, .so, .bin

Note: some PDFs can't be processed as they are image-type files

Exclusion patterns - advanced filtering

You can specify URLs to exclude using patterns. Here are practical examples:

Exclude entire sections

https://theta.co.nz/admin*

This will skip:

Exclude by content type

*login*

This will skip any URL containing "login":

Exclude specific pages

https://theta.co.nz/private
*contact*
*/temp/*

This will skip:

https://theta.co.nz/private (exact match)
https://theta.co.nz/contact-us (contains "contact")
https://theta.co.nz/files/temp/draft.html (contains "/temp/")

Practical examples

Example 1: Company website scan

Starting URL: https://mycompany.com

What Gets Scanned:

Homepage content and navigation
About us, products, services pages
FAQ and help sections
Product documentation (PDFs)
All sub-pages and nested content

What Gets Excluded:

Customer login portal
Admin dashboard
External social media links
Any other external links
Image and video files

Example 2: Documentation site scan

Starting URL: https://docs.myapi.com/v2/

What Gets Scanned:

All v2 API documentation
Code examples and tutorials
Reference guides
Getting started guides

Exclusions You Might Add:

https://docs.myapi.com/v1/*    (exclude old version)
*deprecated*                  (exclude deprecated content)
*/examples/temp/*            (exclude temporary examples)

Example 3: Support portal scan

Starting URL: https://support.theta.co.nz

What gets scanned:

All help articles
Troubleshooting guides
FAQ sections
Tutorial videos (text descriptions)
Downloadable manuals (PDFs)

Common exclusions:

*ticket*          (exclude ticketing system pages)
*private*         (exclude private customer content)
*/admin/*         (exclude admin tools)

Scanning limits

Free trial accounts

Maximum 50 pages per scan
If limit is reached, scanning stops and you're notified
You can run another scan to continue from where it left off

Regular accounts

Higher limits (varies by plan)
Can process hundreds or thousands of pages
Automatic retry if temporary issues occur

How the system handles problems

Automatic recovery

Slow loading pages: Tries different methods to determine when page is fully loaded
Temporary errors: Automatically retries failed pages up to 3 times
Blocked pages: Skips blocked content and continues with other pages
Large websites: Processes pages in batches to avoid overwhelming the system

What you see during scanning

Status updates: Pending → Processing → Completed
Progress tracking: "Processed 25 of 45 pages found"
Final summary: Total pages processed, files uploaded

Note: If you are scanning multiple URLs, even if it says "Scanning...", a page may not yet have started processing

Recurring scans

You can set up automatic scanning to keep your knowledge base up-to-date:

Weekly scans: Automatically re-scan your URLs every week
Change detection: Only updates content that has actually changed

Best practices

Choosing starting URLs

There are two ways of scanning, scan the full site - recommended, or scanning sub sites

Scan full site

This is the recommended approach for most scenarios as the scanning tool will find all sub links from the main site, giving the broadest set of information available to answer questions. Specific sections can be excluded e.g. */financial-reports/* to exclude annual financial reports. Or use this technique to exclude other areas that would not be useful in answering users' questions.

Scan subsites

This is best used when your bot only answers a subset of the questions available. An example of this might be in a larger organisation where the bot is used for a specific purpose. e.g. a council who wants to use the bot to answer questions about an upcoming annual plan.

In this case, provide multiple links and each will be scanned independently. Only sub-sites of these links will be included in the scan.

You may also choose to run your first scan as a sub-site scan, to try it out with a more tightly bounded scope.

Setting up exclusions

Be specific: Use exact URLs when possible
Use patterns wisely: *admin* catches more than */admin/*
Test exclusions: Verify your patterns work with example URLs

Monitoring results

Check the URLs found: Review what's been scanned
Verify content: Spot-check that important pages were captured
Adjust as needed: Refine your URLs and exclusions based on results