`langchain.evaluation.agents.trajectory_eval_chain`.TrajectoryEvalChain¶

class langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain(*, memory: Optional[BaseMemory] = None, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, callback_manager: Optional[BaseCallbackManager] = None, verbose: bool = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, agent_tools: Optional[List[BaseTool]] = None, eval_chain: LLMChain, output_parser: TrajectoryOutputParser = None, return_reasoning: bool = False)[source]¶

Bases: AgentTrajectoryEvaluator, LLMEvalChain

A chain for evaluating ReAct style agents.

This chain is used to evaluate ReAct style agents by reasoning about the sequence of actions taken and their outcomes.

Example:

from langchain.agents import AgentType, initialize_agent
from langchain.chat_models import ChatOpenAI
from langchain.evaluation import TrajectoryEvalChain
from langchain.tools import tool

@tool
def geography_answers(country: str, question: str) -> str:
    """Very helpful answers to geography questions."""
    return f"{country}? IDK - We may never know {question}."

llm = ChatOpenAI(model="gpt-3.5-turbo", temperature=0)
agent = initialize_agent(
    tools=[geography_answers],
    llm=llm,
    agent=AgentType.OPENAI_FUNCTIONS,
    return_intermediate_steps=True,
)

question = "How many dwell in the largest minor region in Argentina?"
response = agent(question)

eval_chain = TrajectoryEvalChain.from_llm(
    llm=llm, agent_tools=[geography_answers], return_reasoning=True
)

result = eval_chain.evaluate_agent_trajectory(
    input=question,
    agent_trajectory=response["intermediate_steps"],
    prediction=response["output"],
    reference="Paris",
)
print(result["score"])
# 0

param agent_tools: Optional[List[langchain.tools.base.BaseTool]] = None¶: A list of tools available to the agent.

param callback_manager: Optional[BaseCallbackManager] = None¶: Deprecated, use callbacks instead.

param callbacks: Callbacks = None¶: Optional list of callback handlers (or callback manager). Defaults to None. Callback handlers are called throughout the lifecycle of a call to a chain, starting with on_chain_start, ending with on_chain_end or on_chain_error. Each custom chain can optionally call additional callback methods, see Callback docs for full details.

param eval_chain: langchain.chains.llm.LLMChain [Required]¶: The language model chain used for evaluation.

param memory: Optional[BaseMemory] = None¶: Optional memory object. Defaults to None. Memory is a class that gets called at the start and at the end of every chain. At the start, memory loads variables and passes them along in the chain. At the end, it saves any returned variables. There are many different types of memory - please see memory docs for the full catalog.

param metadata: Optional[Dict[str, Any]] = None¶: Optional metadata associated with the chain. Defaults to None. This metadata will be associated with each call to this chain, and passed as arguments to the handlers defined in callbacks. You can use these to eg identify a specific instance of a chain with its use case.

param output_parser: langchain.evaluation.agents.trajectory_eval_chain.TrajectoryOutputParser [Optional]¶: The output parser used to parse the output.

param return_reasoning: bool = False¶: DEPRECATED. Reasoning always returned.

param tags: Optional[List[str]] = None¶: Optional list of tags associated with the chain. Defaults to None. These tags will be associated with each call to this chain, and passed as arguments to the handlers defined in callbacks. You can use these to eg identify a specific instance of a chain with its use case.

param verbose: bool [Optional]¶: Whether or not run in verbose mode. In verbose mode, some intermediate logs will be printed to the console. Defaults to langchain.verbose value.

__call__(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶

Execute the chain.

Parameters

inputs – Dictionary of inputs, or single input if chain expects only one param. Should contain all inputs specified in Chain.input_keys except for inputs that will be set by the chain’s memory.
return_only_outputs – Whether to return only outputs in the response. If True, only new keys generated by this chain will be returned. If False, both input keys and new keys generated by this chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults to False.

Returns

A dict of named outputs. Should contain all outputs specified in: Chain.output_keys.

async acall(inputs: Union[Dict[str, Any], Any], return_only_outputs: bool = False, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, *, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, include_run_info: bool = False) → Dict[str, Any]¶

Asynchronously execute the chain.

Parameters

inputs – Dictionary of inputs, or single input if chain expects only one param. Should contain all inputs specified in Chain.input_keys except for inputs that will be set by the chain’s memory.
return_only_outputs – Whether to return only outputs in the response. If True, only new keys generated by this chain will be returned. If False, both input keys and new keys generated by this chain will be returned. Defaults to False.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
metadata – Optional metadata associated with the chain. Defaults to None
include_run_info – Whether to include run info in the response. Defaults to False.

Returns

A dict of named outputs. Should contain all outputs specified in: Chain.output_keys.

async aevaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶

Asynchronously evaluate a trajectory.

Parameters

prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.

Returns

The evaluation result.

Return type

dict

async ainvoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None) → Dict[str, Any]¶

apply(input_list: List[Dict[str, Any]], callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None) → List[Dict[str, str]]¶: Call the chain on all inputs in the list.

async arun(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Convenience method for executing chain.

The main difference between this method and Chain.__call__ is that this method expects inputs to be passed directly in as positional arguments or keyword arguments, whereas Chain.__call__ expects a single input dictionary with all the inputs

Parameters

*args – If the chain expects a single input, it can be passed in as the sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in directly as keyword arguments.

Returns

The chain output.

Example

# Suppose we have a single-input chain that takes a 'question' string:
await chain.arun("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."

# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
await chain.arun(question=question, context=context)
# -> "The temperature in Boise is..."

dict(**kwargs: Any) → Dict¶

Dictionary representation of chain.

Expects Chain._chain_type property to be implemented and for memory to be: null.

Parameters: **kwargs – Keyword arguments passed to default pydantic.BaseModel.dict method.
Returns: A dictionary representation of the chain.

Example

..code-block:: python

chain.dict(exclude_unset=True) # -> {“_type”: “foo”, “verbose”: False, …}

evaluate_agent_trajectory(*, prediction: str, agent_trajectory: Sequence[Tuple[AgentAction, str]], input: str, reference: Optional[str] = None, **kwargs: Any) → dict¶

Evaluate a trajectory.

Parameters

prediction (str) – The final predicted response.
agent_trajectory (List[Tuple[AgentAction, str]]) – The intermediate steps forming the agent trajectory.
input (str) – The input to the agent.
reference (Optional[str]) – The reference answer.

Returns

The evaluation result.

Return type

dict

classmethod from_llm(llm: BaseLanguageModel, agent_tools: Optional[Sequence[BaseTool]] = None, output_parser: Optional[TrajectoryOutputParser] = None, **kwargs: Any) → TrajectoryEvalChain[source]¶

Create a TrajectoryEvalChain object from a language model chain.

Parameters

llm (BaseChatModel) – The language model chain.
agent_tools (Optional[Sequence[BaseTool]]) – A list of tools available to the agent.
output_parser (Optional[TrajectoryOutputParser]) – The output parser used to parse the chain output into a score.

Returns

The TrajectoryEvalChain object.

Return type

TrajectoryEvalChain

static get_agent_trajectory(steps: Union[str, Sequence[Tuple[AgentAction, str]]]) → str[source]¶

Get the agent trajectory as a formatted string.

Parameters: steps (Union[str, List[Tuple[AgentAction, str]]]) – The agent trajectory.
Returns: The formatted agent trajectory.
Return type: str

invoke(input: Dict[str, Any], config: Optional[RunnableConfig] = None) → Dict[str, Any]¶

prep_inputs(inputs: Union[Dict[str, Any], Any]) → Dict[str, str][source]¶: Validate and prep inputs.

prep_outputs(inputs: Dict[str, str], outputs: Dict[str, str], return_only_outputs: bool = False) → Dict[str, str]¶

Validate and prepare chain outputs, and save info about this run to memory.

Parameters

inputs – Dictionary of chain inputs, including any inputs added by chain memory.
outputs – Dictionary of initial chain outputs.
return_only_outputs – Whether to only return the chain outputs. If False, inputs are also added to the final outputs.

Returns

A dict of the final chain outputs.

validator raise_callback_manager_deprecation » all fields¶: Raise deprecation warning if callback_manager is used.

run(*args: Any, callbacks: Optional[Union[List[BaseCallbackHandler], BaseCallbackManager]] = None, tags: Optional[List[str]] = None, metadata: Optional[Dict[str, Any]] = None, **kwargs: Any) → Any¶

Convenience method for executing chain.

Parameters

*args – If the chain expects a single input, it can be passed in as the sole positional argument.
callbacks – Callbacks to use for this chain run. These will be called in addition to callbacks passed to the chain during construction, but only these runtime callbacks will propagate to calls to other objects.
tags – List of string tags to pass to all callbacks. These will be passed in addition to tags passed to the chain during construction, but only these runtime tags will propagate to calls to other objects.
**kwargs – If the chain expects multiple inputs, they can be passed in directly as keyword arguments.

Returns

The chain output.

Example

# Suppose we have a single-input chain that takes a 'question' string:
chain.run("What's the temperature in Boise, Idaho?")
# -> "The temperature in Boise is..."

# Suppose we have a multi-input chain that takes a 'question' string
# and 'context' string:
question = "What's the temperature in Boise, Idaho?"
context = "Weather report for Boise, Idaho on 07/03/23..."
chain.run(question=question, context=context)
# -> "The temperature in Boise is..."

save(file_path: Union[Path, str]) → None¶

Save the chain.

Expects Chain._chain_type property to be implemented and for memory to be: null.

Parameters: file_path – Path to file to save the chain to.

Example

chain.save(file_path="path/chain.yaml")

validator set_verbose » verbose¶

Set the chain verbosity.

Defaults to the global setting if not specified by the user.

to_json() → Union[SerializedConstructor, SerializedNotImplemented]¶

to_json_not_implemented() → SerializedNotImplemented¶

property input_keys: List[str]¶

Get the input keys for the chain.

Returns: The input keys.
Return type: List[str]

property lc_attributes: Dict¶: Return a list of attribute names that should be included in the serialized kwargs. These attributes must be accepted by the constructor.

property lc_namespace: List[str]¶: Return the namespace of the langchain object. eg. [“langchain”, “llms”, “openai”]

property lc_secrets: Dict[str, str]¶: Return a map of constructor argument names to secret ids. eg. {“openai_api_key”: “OPENAI_API_KEY”}

property lc_serializable: bool¶: Return whether or not the class is serializable.

property output_keys: List[str]¶

Get the output keys for the chain.

Returns: The output keys.
Return type: List[str]

property requires_input: bool¶: Whether this evaluator requires an input string.

property requires_reference: bool¶: Whether this evaluator requires a reference label.

model Config[source]¶

Bases: object

Configuration for the QAEvalChain.

extra = 'ignore'¶

langchain.evaluation.agents.trajectory_eval_chain.TrajectoryEvalChain¶

`langchain.evaluation.agents.trajectory_eval_chain`.TrajectoryEvalChain¶