langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader¶

class langchain.document_loaders.recursive_url_loader.RecursiveUrlLoader(url: str, exclude_dirs: Optional[str] = None)[source]¶

Bases: BaseLoader

Loads all child links from a given url.

Initialize with URL to crawl and any subdirectories to exclude.

Parameters

  • url – The URL to crawl.

  • exclude_dirs – A list of subdirectories to exclude.

Methods

__init__(url[, exclude_dirs])

Initialize with URL to crawl and any subdirectories to exclude.

get_child_links_recursive(url[, visited])

Recursively get all child links starting with the path of the input URL.

lazy_load()

Lazy load web pages.

load()

Load web pages.

load_and_split([text_splitter])

Load Documents and split into chunks.

Recursively get all child links starting with the path of the input URL.

Parameters

  • url – The URL to crawl.

  • visited – A set of visited URLs.

lazy_load() → Iterator[Document][source]¶

Lazy load web pages.

load() → List[Document][source]¶

Load web pages.

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

Examples using RecursiveUrlLoader¶