Interface WEBCRAWLERConfig

Additional Allowed URLs or prefix(es). Add one or more allowed URLs or URL prefixes. The crawler will read URLs that match these patterns in addition to the seed URL(s).. Example: (e.g. https://docs.example.com)

Forbidden Paths. Example: Enter forbidden paths (e.g. /admin)

Max Depth. Example: Enter maximum crawl depth

Max Error Count. Example: Enter maximum error count

Max URLs. Example: Enter maximum number of URLs to crawl

Throttle (ms). Example: Enter minimum time between requests in milliseconds

Reindex Interval (seconds). Example: Enter reindex interval in seconds