langchain.document_loaders.apify_dataset.ApifyDatasetLoader¶

class langchain.document_loaders.apify_dataset.ApifyDatasetLoader(dataset_id: str, dataset_mapping_function: Callable[[Dict], Document])[source]¶

Bases: BaseLoader, BaseModel

Loads datasets from Apify-a web scraping, crawling, and data extraction platform. For details, see https://docs.apify.com/platform/integrations/langchain

Example

from langchain.document_loaders import ApifyDatasetLoader
from langchain.schema import Document

loader = ApifyDatasetLoader(
    dataset_id="YOUR-DATASET-ID",
    dataset_mapping_function=lambda dataset_item: Document(
        page_content=dataset_item["text"], metadata={"source": dataset_item["url"]}
    ),
)
documents = loader.load()

Initialize the loader with an Apify dataset ID and a mapping function.

Parameters

  • dataset_id (str) – The ID of the dataset on the Apify platform.

  • dataset_mapping_function (Callable) – A function that takes a single dictionary (an Apify dataset item) and converts it to an instance of the Document class.

param apify_client: Any = None¶

An instance of the ApifyClient class from the apify-client Python package.

param dataset_id: str [Required]¶

The ID of the dataset on the Apify platform.

param dataset_mapping_function: Callable[[Dict], langchain.schema.document.Document] [Required]¶

A custom function that takes a single dictionary (an Apify dataset item) and converts it to an instance of the Document class.

lazy_load() → Iterator[Document]¶

A lazy loader for Documents.

load() → List[Document][source]¶

Load documents.

load_and_split(text_splitter: Optional[TextSplitter] = None) → List[Document]¶

Load Documents and split into chunks. Chunks are returned as Documents.

Parameters

text_splitter – TextSplitter instance to use for splitting documents. Defaults to RecursiveCharacterTextSplitter.

Returns

List of Documents.

validator validate_environment  »  all fields[source]¶

Validate environment.

Parameters

values – The values to validate.

Examples using ApifyDatasetLoader¶