`langchain.evaluation.schema`.PairwiseStringEvaluator¶

class langchain.evaluation.schema.PairwiseStringEvaluator[source]¶

Bases: _EvalArgsMixin, ABC

Compare the output of two models (or two outputs of the same model).

Methods

`__init__`()
`aevaluate_string_pairs`(*, prediction, ...[, ...])	Asynchronously evaluate the output string pairs.
`evaluate_string_pairs`(*, prediction, ...[, ...])	Evaluate the output string pairs.

Attributes

`requires_input`	Whether this evaluator requires an input string.
`requires_reference`	Whether this evaluator requires a reference label.

async aevaluate_string_pairs(*, prediction: str, prediction_b: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶

Asynchronously evaluate the output string pairs.

Parameters

prediction (str) – The output string from the first model.
prediction_b (str) – The output string from the second model.
reference (Optional[str], optional) – The expected output / reference string.
input (Optional[str], optional) – The input string.
**kwargs – Additional keyword arguments, such as callbacks and optional reference strings.

Returns

A dictionary containing the preference, scores, and/or other information.

Return type

dict

evaluate_string_pairs(*, prediction: str, prediction_b: str, reference: Optional[str] = None, input: Optional[str] = None, **kwargs: Any) → dict[source]¶

Evaluate the output string pairs.