langchain.document_loaders.hn.HNLoader

class langchain.document_loaders.hn.HNLoader(web_path: Union[str, List[str]], header_template: Optional[dict] = None, verify_ssl: Optional[bool] = True, proxies: Optional[dict] = None)

Bases: WebBaseLoader

Load Hacker News data from either main page results or the comments page.

Initialize with webpage path.

Methods

__init__(web_path[, header_template, ...])	Initialize with webpage path.
aload()	Load text from the urls in web_path async into Documents.
fetch_all(urls)	Fetch all urls concurrently with rate limiting.
lazy_load()	Lazy load text from the url(s) in web_path.
load()	Get important HN webpage information.
load_and_split([text_splitter])	Load Documents and split into chunks.
load_comments(soup_info)	Load comments from a HN post.
load_results(soup)	Load items from an HN page.
scrape([parser])	Scrape data from webpage and return it in BeautifulSoup format.
scrape_all(urls[, parser])	Fetch all urls, then return soups for all results.

Attributes

bs_get_text_kwargs	kwargs for beatifulsoup4 get_text
default_parser	Default parser to use for BeautifulSoup.
raise_for_status	Raise an exception if http status code denotes an error.
requests_kwargs	kwargs for requests
requests_per_second	Max number of concurrent requests to make.
web_path

aload() → List[Document]

Load text from the urls in web_path async into Documents.

async fetch_all(urls: List[str]) → Any

Fetch all urls concurrently with rate limiting.

lazy_load() → Iterator[Document]

Lazy load text from the url(s) in web_path.

load() → List[Document]

Get important HN webpage information.

HN webpage components are:

• title

• content

• source url,

• time of post

• author of the post

• number of comments

• rank of the post

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

load_comments(soup_info: Any) → List[Document]

Load comments from a HN post.

load_results(soup: Any) → List[Document]

Load items from an HN page.

scrape(parser: Optional[str] = None) → Any

Scrape data from webpage and return it in BeautifulSoup format.

scrape_all(urls: List[str], parser: Optional[str] = None) → List[Any]

Fetch all urls, then return soups for all results.

bs_get_text_kwargs: Dict[str, Any] = {}

kwargs for beatifulsoup4 get_text

default_parser: str = 'html.parser'

Default parser to use for BeautifulSoup.

raise_for_status: bool = False

Raise an exception if http status code denotes an error.

requests_kwargs: Dict[str, Any] = {}

kwargs for requests

requests_per_second: int = 2

Max number of concurrent requests to make.

property web_path: str

web_paths: List[str]

Examples using HNLoader

• Hacker News