`langchain.evaluation.string_distance.base`.StringDistanceEvalChain¶

class langchain.evaluation.string_distance.base.StringDistanceEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, distance: StringDistance = StringDistance.JARO_WINKLER, normalize_score: bool = True)[source]¶

Bases: StringEvaluator, _RapidFuzzChainMixin

Compute string distances between the prediction and the reference.

Examples

>>> from langchain.evaluation import StringDistanceEvalChain
>>> evaluator = StringDistanceEvalChain()
>>> evaluator.evaluate_strings(
        prediction="Mindy is the CTO",
        reference="Mindy is the CEO",
    )

Using the load_evaluator function:

>>> from langchain.evaluation import load_evaluator
>>> evaluator = load_evaluator("string_distance")
>>> evaluator.evaluate_strings(
        prediction="The answer is three",
        reference="three",
    )

param callback_manager: Optional[BaseCallbackManager] = None¶: Deprecated, use callbacks instead.

param callbacks: Callbacks = None¶: Optional list of callback handlers (or callback manager). Defaults to None. Callback handlers are called throughout the lifecycle of a call to a chain, starting with on_chain_start, ending with on_chain_end or on_chain_error. Each custom chain can optionally call additional callback methods, see Callback docs for full details.

param distance: StringDistance = StringDistance.JARO_WINKLER¶

param memory: Optional[BaseMemory] = None¶: Optional memory object. Defaults to None. Memory is a class that gets called at the start and at the end of every chain. At the start, memory loads variables and passes them along in the chain. At the end, it saves any returned variables. There are many different types of memory - please see memory docs for the full catalog.

param metadata: Optional[Dict[str, Any]] = None¶: Optional metadata associated with the chain. Defaults to None. This metadata will be associated with each call to this chain, and passed as arguments to the handlers defined in callbacks. You can use these to eg identify a specific instance of a chain with its use case.

param normalize_score: bool = True¶: Whether to normalize the score to a value between 0 and 1. Applies only to the Levenshtein and Damerau-Levenshtein distances.

param tags: Optional[List[str]] = None¶: Optional list of tags associated with the chain. Defaults to None. These tags will be associated with each call to this chain, and passed as arguments to the handlers defined in callbacks. You can use these to eg identify a specific instance of a chain with its use case.

param verbose: bool [Optional]¶: Whether or not run in verbose mode. In verbose mode, some intermediate logs will be printed to the console. Defaults to langchain.verbose value.

__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶

Execute the chain.

Parameters

inputs – Dictionary of inputs, or single input if chain expects only one param. Should contain all inputs specified in Chain.input_keys except for inputs that will be set by the chain’s memory.
return_only_outputs – Whether to return only outputs in the response. If True, only new keys generated by this chain will be returned. If False, both input keys and new keys generated by this chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults to False.

Returns

A dict of named outputs. Should contain all outputs specified in: Chain.output_keys.

async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶

Asynchronously execute the chain.

Parameters

inputs – Dictionary of inputs, or single input if chain expects only one param. Should contain all inputs specified in Chain.input_keys except for inputs that will be set by the chain’s memory.
return_only_outputs – Whether to return only outputs in the response. If True, only new keys generated by this chain will be returned. If False, both input keys and new keys generated by this chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults to False.

Returns

A dict of named outputs. Should contain all outputs specified in: Chain.output_keys.

async aevaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶

Asynchronously evaluate Chain or LLM output, based on optional input and label.

Parameters

prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.

Returns

The evaluation results containing the score or value.

Return type

dict

async ainvoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None) → Dict[str, Any]¶

apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶: Call the chain on all inputs in the list.

async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Convenience method for executing chain.

The main difference between this method and Chain.__call__ is that this method expects inputs to be passed directly in as positional arguments or keyword arguments, whereas Chain.__call__ expects a single input dictionary with all the inputs

Parameters

*args – If the chain expects a single input, it can be passed in as the sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in directly as keyword arguments.

Returns

The chain output.

Example

# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."

# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."

compute_metric(a: str, b: str) → float¶

Compute the distance between two strings.

Parameters

a (str) – The first string.
b (str) – The second string.

Returns

The distance between the two strings.

Return type

float

dict(**kwargs: Any) → Dict¶

Dictionary representation of chain.

Expects Chain._chain_type property to be implemented and for memory to be: null.

Parameters: **kwargs – Keyword arguments passed to default pydantic.BaseModel.dict method.
Returns: A dictionary representation of the chain.

Example

..code-block:: python

chain.dict(exclude_unset=True) # -> {“_type”: “foo”, “verbose”: False, …}

evaluate_strings(*, prediction: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict¶

Evaluate Chain or LLM output, based on optional input and label.

Parameters

prediction (str) – The LLM or chain prediction to evaluate.
reference (Optional[str], optional) – The reference label to evaluate against.
input (Optional[str], optional) – The input to consider during evaluation.
**kwargs – Additional keyword arguments, including callbacks, tags, etc.

Returns

The evaluation results containing the score or value.

Return type

dict

invoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None) → Dict[str, Any]¶

prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str]¶

Validate and prepare chain inputs, including adding inputs from memory.

Parameters: inputs – Dictionary of raw inputs, or single input if chain expects only one param. Should contain all inputs specified in Chain.input_keys except for inputs that will be set by the chain’s memory.
Returns: A dictionary of all inputs, including those added by the chain’s memory.

prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶

Validate and prepare chain outputs, and save info about this run to memory.

Parameters

inputs – Dictionary of chain inputs, including any inputs added by chain memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False, inputs are also added to the final outputs.

Returns

A dict of the final chain outputs.

validator raise_callback_manager_deprecation » all fields¶: Raise deprecation warning if callback_manager is used.

run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Convenience method for executing chain.

Parameters

*args – If the chain expects a single input, it can be passed in as the sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in directly as keyword arguments.

Returns

The chain output.

Example

# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."

# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."

save(file_path: Union[Path, str]) → None¶

Save the chain.

Expects Chain._chain_type property to be implemented and for memory to be: null.

Parameters: file_path – Path to file to save the chain to.

Example

chain.save(file_path="path/chain.yaml")

validator set_verbose » verbose¶

Set the chain verbosity.

Defaults to the global setting if not specified by the user.

to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶

to_json_not_implemented() → SerializedNotImplemented¶

validator validate_dependencies » all fields¶

Validate that the rapidfuzz library is installed.

Parameters: values (Dict[str, Any]) – The input values.
Returns: The validated values.
Return type: Dict[str, Any]

property evaluation_name: str¶

Get the evaluation name.

Returns: The evaluation name.
Return type: str

property input_keys: List[str]¶

Get the input keys.

Returns: The input keys.
Return type: List[str]

property lc_attributes: Dict¶: Return a list of attribute names that should be included in the serialized kwargs. These attributes must be accepted by the constructor.

property lc_namespace: List[str]¶: Return the namespace of the langchain object. eg. [“langchain”, “llms”, “openai”]

property lc_secrets: Dict[str, str]¶: Return a map of constructor argument names to secret ids. eg. {“openai_api_key”: “OPENAI_API_KEY”}

property lc_serializable: bool¶: Return whether or not the class is serializable.

property metric: Callable¶

Get the distance metric function.

Returns: The distance metric function.
Return type: Callable

property output_keys: List[str]¶

Get the output keys.

Returns: The output keys.
Return type: List[str]

property requires_input: bool¶: This evaluator does not require input.

property requires_reference: bool¶: This evaluator does not require a reference.

model Config¶

Bases: object

Configuration for this pydantic object.

arbitrary_types_allowed = True¶

langchain.evaluation.string_distance.base.StringDistanceEvalChain¶

`langchain.evaluation.string_distance.base`.StringDistanceEvalChain¶