langchain.document_loaders.gitbook.GitbookLoader

class langchain.document_loaders.gitbook.GitbookLoader(web_page: str, load_all_paths: bool = False, base_url: Optional[str] = None, content_selector: str = 'main')

Bases: WebBaseLoader

Load GitBook data.

1. load from either a single page, or

2. load all (relative) paths in the navbar.

Initialize with web page and whether to load all paths.

Parameters

• web_page – The web page to load or the starting point from where relative paths are discovered.

• load_all_paths – If set to True, all relative paths in the navbar are loaded instead of only web_page.

• base_url – If load_all_paths is True, the relative paths are appended to this base url. Defaults to web_page.

• content_selector – The CSS selector for the content to load. Defaults to “main”.

Methods

__init__(web_page[, load_all_paths, ...])	Initialize with web page and whether to load all paths.
aload()	Load text from the urls in web_path async into Documents.
fetch_all(urls)	Fetch all urls concurrently with rate limiting.
lazy_load()	Lazy load text from the url(s) in web_path.
load()	Fetch text from one single GitBook page.
load_and_split([text_splitter])	Load Documents and split into chunks.
scrape([parser])	Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])	Fetch all urls, then return soups for all results.

Attributes

bs_get_text_kwargs	kwargs for beatifulsoup4 get_text
default_parser	Default parser to use for BeautifulSoup.
raise_for_status	Raise an exception if http status code denotes an error.
requests_kwargs	kwargs for requests
requests_per_second	Max number of concurrent requests to make.
web_path

aload() → List[Document]

Load text from the urls in web_path async into Documents.

async fetch_all(urls: List[str]) → Any

Fetch all urls concurrently with rate limiting.

lazy_load() → Iterator[Document]

Lazy load text from the url(s) in web_path.

load() → List[Document]

Fetch text from one single GitBook page.

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

scrape(parser: Optional[str] = None) → Any

Scrape data from webpage and return it in BeautifulSoup format.

scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]

Fetch all urls, then return soups for all results.

bs_get_text_kwargs: Dict[str, Any] = {}

kwargs for beatifulsoup4 get_text

default_parser: str = 'html.parser'

Default parser to use for BeautifulSoup.

raise_for_status: bool = False

Raise an exception if http status code denotes an error.

requests_kwargs: Dict[str, Any] = {}

kwargs for requests

requests_per_second: int = 2

Max number of concurrent requests to make.

property web_path: str

web_paths: List[str]

Examples using GitbookLoader

• GitBook