{"versionId":"f6ddf71d-5906-4ba3-19a6-08de2ad71e6a","articleId":"a42b7aa5-41f1-4628-84af-08de1cbee49d","title":"Assistant answers: URL scanning ","tags":["GPT","All articles"],"content":"

URL scanning allows you to automatically discover and index content from websites by simply providing a list of starting URLs. 

\n

For example, if you paste in https://theta.co.nz the system will:

\n
    \n
  1. Scan that page for content
    2. \n
  2. Find all links on that page
    3. \n
  3. Follow those links (if they're on the same website)
    4. \n
  4. Continue this process to build a comprehensive knowledge base, which can then be used by your chatbot's AI assistant to help answer user questions.
    5. \n
\n
\n

In this article

\n\n
\n

How it works

\n

Starting the scan

\n

You provide a list of URLs to scan, one per line, for example:

\n\n

The system will process each URL and automatically discover related content.

\n

Automatic content discovery

\n

When scanning https://theta.co.nz/products the system will:

\n
    \n
  1. Extract the page content - All text, headings, and useful information
    2. \n
  2. Find all links - Every clickable link on the page
    3. \n
  3. Follow relevant links - Only links that meet the inclusion rules (see below)
    4. \n
  4. Repeat the process - Continue scanning newly discovered pages
    5. \n
\n

What gets included vs excluded

\n

✅ Pages that get scanned

\n

Same domain pages If you start with https://theta.co.nz, these will be included:

\n\n

Sub-path pages If you start with https://theta.co.nz/support, these will be included:

\n\n

Document files

\n\n

Processable extensions (downloaded and indexed):

\n\n

❌ Pages that get ignored

\n

Different domains 

\n

If you start with https://theta.co.nz, these will be ignored:

\n\n

Parent Paths 

\n

If you start with https://theta.co.nz/support/guides, these will be ignored:

\n\n

Binary Files

\n\n

Excluded Extensions (skipped during crawling):

\n\n

Note: some PDFs can't be processed as they are image-type files

\n

Exclusion patterns - advanced filtering

\n

You can specify URLs to exclude using patterns. Here are practical examples:

\n

Exclude entire sections

\n
https://theta.co.nz/admin*\n\n
\n

This will skip:

\n\n

Exclude by content type

\n
*login*\n\n
\n

This will skip any URL containing \"login\":

\n\n

Exclude specific pages

\n
https://theta.co.nz/private\n*contact*\n*/temp/*\n\n
\n

This will skip:

\n\n

Practical examples

\n

Example 1: Company website scan

\n

Starting URL: https://mycompany.com

\n

What Gets Scanned:

\n\n

What Gets Excluded:

\n\n

Example 2: Documentation site scan

\n

Starting URL: https://docs.myapi.com/v2/

\n

What Gets Scanned:

\n\n

Exclusions You Might Add:

\n
https://docs.myapi.com/v1/*    (exclude old version)\n*deprecated*                  (exclude deprecated content)\n*/examples/temp/*            (exclude temporary examples)\n\n
\n

Example 3: Support portal scan

\n

Starting URL: https://support.theta.co.nz

\n

What gets scanned:

\n\n

Common exclusions:

\n
*ticket*          (exclude ticketing system pages)\n*private*         (exclude private customer content)\n*/admin/*         (exclude admin tools)\n\n
\n

Scanning limits

\n

Free trial accounts

\n\n

Regular accounts

\n\n

How the system handles problems

\n

Automatic recovery

\n\n

What you see during scanning

\n\n

Note: If you are scanning multiple URLs, even if it says \"Scanning...\", a page may not yet have started processing

\n

 

\n
 
\n

Recurring scans

\n

You can set up automatic scanning to keep your knowledge base up-to-date:

\n\n

Best practices

\n

Choosing starting URLs

\n

There are two ways of scanning, scan the full site - recommended, or scanning sub sites

\n

Scan full site

\n

This is the recommended approach for most scenarios as the scanning tool will find all sub links from the main site, giving the broadest set of information available to answer questions. Specific sections can be excluded e.g. */financial-reports/* to exclude annual financial reports. Or use this technique to exclude other areas that would not be useful in answering users' questions.

\n

Scan subsites

\n

This is best used when your bot only answers a subset of the questions available. An example of this might be in a larger organisation where the bot is used for a specific purpose. e.g. a council who wants to use the bot to answer questions about an upcoming annual plan.

\n

In this case, provide multiple links and each will be scanned independently. Only sub-sites of these links will be included in the scan.

\n

You may also choose to run your first scan as a sub-site scan, to try it out with a more tightly bounded scope. 

\n

Setting up exclusions

\n\n

Monitoring results

\n\n

 

","description":"","isMarkdown":false,"publishedOnUtc":"2025-11-25T03:04:45.5748639","modifiedOnUtc":"2025-11-05T22:58:53.6725147"}