Helpfruit - Web Crawler URL Scanning Feature - User Guide

Overview

The Web Crawler URL Scanning feature allows you to automatically discover and index content from websites by simply providing a list of starting URLs. For example, if you paste in https://theta.co.nz the system will:

Scan that page for content
Find all links on that page
Follow those links (if they're on the same website)
Continue this process to build a comprehensive knowledge base

How It Works

Starting the Scan

You provide a list of URLs to scan, one per line:

https://theta.co.nz
https://support.theta.co.nz/getting-started
https://docs.theta.co.nz/api

The system will process each URL and automatically discover related content.

Automatic Content Discovery

When scanning https://theta.co.nz/products, the system will:

Extract the page content - All text, headings, and useful information
Find all links - Every clickable link on the page
Follow relevant links - Only links that meet the inclusion rules
Repeat the process - Continue scanning newly discovered pages

What Gets Included vs Excluded

✅ Pages That Get Scanned

Same Domain Pages If you start with https://theta.co.nz, these will be included:

Sub-Path Pages If you start with https://theta.co.nz/support, these will be included:

Document Files

PDF files: https://theta.co.nz/manual.pdf
Word documents: [https://theta.co.nz/guide.docx](https://theta.co.nz/guide.docx)
Text files: [https://theta.co.nz/changelog.txt](https://theta.co.nz/changelog.txt)

Processable Extensions (downloaded and indexed):

Documents: .pdf, .doc, .docx, .txt
Web content: .html, .htm

❌ Pages That Get Ignored

Different Domains If you start with [https://theta.co.nz](https://theta.co.nz), these will be ignored:

https://microsoft.com (completely different domain)
https://support.google.com (different domain)
https://dash.theta.co.nz (different subdomain)

Parent Paths If you start with https://theta.co.nz/support/guides, these will be ignored:

https://theta.co.nz/about (above the starting path)
https://theta.co.nz/products (above the starting path)
[https://theta.co.nz/manual.pdf] (above the starting path)

Binary Files

Images: https://theta.co.nz/logo.png
Videos: https://theta.co.nz/demo.mp4
Software: https://theta.co.nz/installer.exe
Archives: https://theta.co.nz/backup.zip

Excluded Extensions (skipped during crawling):

Images: .png, .jpg, .jpeg, .gif, .bmp, .tiff, .ico, .svg
Spreadsheets: .xls, .xlsx
Archives: .zip, .rar, .gz, .tar, .7z
Media: .mp3, .mp4, .avi, .mov, .flv, .wmv
Executables: .exe, .dll, .so, .bin

Note: some PDFs can't be processed as they are image-type files

Exclusion Patterns - Advanced Filtering

You can specify URLs to exclude using patterns. Here are practical examples:

Exclude Entire Sections

https://theta.co.nz/admin*

This will skip:

Exclude by Content Type

*login*

This will skip any URL containing "login":

Exclude Specific Pages

https://theta.co.nz/private
*contact*
*/temp/*

This will skip:

https://theta.co.nz/private (exact match)
https://theta.co.nz/contact-us (contains "contact")
https://theta.co.nz/files/temp/draft.html (contains "/temp/")

Practical Examples

Example 1: Company Website Scan

Starting URL: https://mycompany.com

What Gets Scanned:

Homepage content and navigation
About us, products, services pages
FAQ and help sections
Product documentation (PDFs)
All sub-pages and nested content

What Gets Excluded:

Customer login portal
Admin dashboard
External social media links
Any other external links
Image and video files

Example 2: Documentation Site Scan

Starting URL: https://docs.myapi.com/v2/

What Gets Scanned:

All v2 API documentation
Code examples and tutorials
Reference guides
Getting started guides

Exclusions You Might Add:

https://docs.myapi.com/v1/*    (exclude old version)
*deprecated*                  (exclude deprecated content)
*/examples/temp/*            (exclude temporary examples)

Example 3: Support Portal Scan

Starting URL: https://support.theta.co.nz

What Gets Scanned:

All help articles
Troubleshooting guides
FAQ sections
Tutorial videos (text descriptions)
Downloadable manuals (PDFs)

Common Exclusions:

*ticket*          (exclude ticketing system pages)
*private*         (exclude private customer content)
*/admin/*         (exclude admin tools)

Scanning Limits

Free Trial Accounts

Maximum 50 pages per scan
If limit is reached, scanning stops and you're notified
You can run another scan to continue from where it left off

Regular Accounts

Higher limits (varies by plan)
Can process hundreds or thousands of pages
Automatic retry if temporary issues occur

How the System Handles Problems

Automatic Recovery

Slow loading pages: Tries different methods to determine when page is fully loaded
Temporary errors: Automatically retries failed pages up to 3 times
Blocked pages: Skips blocked content and continues with other pages
Large websites: Processes pages in batches to avoid overwhelming the system

What You See During Scanning

Status updates: Pending → Processing → Completed
Progress tracking: "Processed 25 of 45 pages found"
Detailed logs: Shows which pages succeeded/failed and why - sys admin users only
Final summary: Total pages processed, files uploaded, any issues encountered - issues shown to sys admin users only

Monitoring Your Scan

Real-time Status

You can see:

Current status (Pending, Scanning, Completed, Error, Cancelled)
How many pages have been found vs. processed
Detailed log of what's happening (Sys Admin only)

Note: If you don't yet see a + icon next to the scanning line, it means that page has not yet been processed (even if it says "Scanning...)

Example Log Messages

[Info] Starting crawl for URL: https://theta.co.nz
[Info] Found 12 links, adding valid links to queue
[Info] Processing file upload for https://theta.co.nz/about
[Info] Uploaded https://theta.co.nz/products.pdf
[Warning] Skipping URL https://theta.co.nz/admin as it is in exclude list
[Info] No change detected for https://theta.co.nz/faq. Skipping upload.
[Info] Found 45 files. Beginning cleanup of old content...

Recurring Scans

You can set up automatic scanning to keep your knowledge base up-to-date:

Weekly scans: Automatically re-scan your URLs every week
Change detection: Only updates content that has actually changed
Load balancing: System spreads scans across different days to avoid overload

Best Practices

Choosing Starting URLs

There are two ways of scanning, scan the full site - recommended, or scanning sub sites

Scan full site

This is the recommended approach for most scenarios as the scanning tool will find all sub links from the main site, giving the broadest set of information available to answer questions. Specific sections can be excluded e.g. */financial-reports/* to exclude annual financial reports. Or use this technique to exclude other areas that would not be useful in answering users' questions.

Scan subsites

This is best used when your bot only answers a subset of the questions available. An example of this might be in a larger organisation where the bot is used for a specific purpose. e.g. a council who wants to use the bot to answer questions about an upcoming annual plan.

In this case, provide multiple links and each will be scanned independently. Only subsites of these links will be included in the scan.

You may also choose to run your first scan as a subsite scan, to try it out with a more tightly bounded scope. Also this approach can be useful for testing.

Setting Up Exclusions

Be specific: Use exact URLs when possible
Use patterns wisely: *admin* catches more than */admin/*
Test exclusions: Verify your patterns work with example URLs

Monitoring Results

Check the logs: Review what was included/excluded
Verify content: Spot-check that important pages were captured
Adjust as needed: Refine your URLs and exclusions based on results

This scanning feature helps to automatically build and maintain comprehensive knowledge bases from website content.