| <html><body> |
| <style> |
| |
| body, h1, h2, h3, div, span, p, pre, a { |
| margin: 0; |
| padding: 0; |
| border: 0; |
| font-weight: inherit; |
| font-style: inherit; |
| font-size: 100%; |
| font-family: inherit; |
| vertical-align: baseline; |
| } |
| |
| body { |
| font-size: 13px; |
| padding: 1em; |
| } |
| |
| h1 { |
| font-size: 26px; |
| margin-bottom: 1em; |
| } |
| |
| h2 { |
| font-size: 24px; |
| margin-bottom: 1em; |
| } |
| |
| h3 { |
| font-size: 20px; |
| margin-bottom: 1em; |
| margin-top: 1em; |
| } |
| |
| pre, code { |
| line-height: 1.5; |
| font-family: Monaco, 'DejaVu Sans Mono', 'Bitstream Vera Sans Mono', 'Lucida Console', monospace; |
| } |
| |
| pre { |
| margin-top: 0.5em; |
| } |
| |
| h1, h2, h3, p { |
| font-family: Arial, sans serif; |
| } |
| |
| h1, h2, h3 { |
| border-bottom: solid #CCC 1px; |
| } |
| |
| .toc_element { |
| margin-top: 0.5em; |
| } |
| |
| .firstline { |
| margin-left: 2 em; |
| } |
| |
| .method { |
| margin-top: 1em; |
| border: solid 1px #CCC; |
| padding: 1em; |
| background: #EEE; |
| } |
| |
| .details { |
| font-weight: bold; |
| font-size: 14px; |
| } |
| |
| </style> |
| |
| <h1><a href="ces_v1beta.html">Gemini Enterprise for Customer Experience API</a> . <a href="ces_v1beta.projects.html">projects</a> . <a href="ces_v1beta.projects.locations.html">locations</a> . <a href="ces_v1beta.projects.locations.apps.html">apps</a> . <a href="ces_v1beta.projects.locations.apps.evaluations.html">evaluations</a> . <a href="ces_v1beta.projects.locations.apps.evaluations.results.html">results</a></h1> |
| <h2>Instance Methods</h2> |
| <p class="toc_element"> |
| <code><a href="#close">close()</a></code></p> |
| <p class="firstline">Close httplib2 connections.</p> |
| <p class="toc_element"> |
| <code><a href="#delete">delete(name, x__xgafv=None)</a></code></p> |
| <p class="firstline">Deletes an evaluation result.</p> |
| <p class="toc_element"> |
| <code><a href="#get">get(name, x__xgafv=None)</a></code></p> |
| <p class="firstline">Gets details of the specified evaluation result.</p> |
| <p class="toc_element"> |
| <code><a href="#list">list(parent, filter=None, orderBy=None, pageSize=None, pageToken=None, x__xgafv=None)</a></code></p> |
| <p class="firstline">Lists all evaluation results for a given evaluation.</p> |
| <p class="toc_element"> |
| <code><a href="#list_next">list_next()</a></code></p> |
| <p class="firstline">Retrieves the next page of results.</p> |
| <h3>Method Details</h3> |
| <div class="method"> |
| <code class="details" id="close">close()</code> |
| <pre>Close httplib2 connections.</pre> |
| </div> |
| |
| <div class="method"> |
| <code class="details" id="delete">delete(name, x__xgafv=None)</code> |
| <pre>Deletes an evaluation result. |
| |
| Args: |
| name: string, Required. The resource name of the evaluation result to delete. (required) |
| x__xgafv: string, V1 error format. |
| Allowed values |
| 1 - v1 error format |
| 2 - v2 error format |
| |
| Returns: |
| An object of the form: |
| |
| { # A generic empty message that you can re-use to avoid defining duplicated empty messages in your APIs. A typical example is to use it as the request or the response type of an API method. For instance: service Foo { rpc Bar(google.protobuf.Empty) returns (google.protobuf.Empty); } |
| }</pre> |
| </div> |
| |
| <div class="method"> |
| <code class="details" id="get">get(name, x__xgafv=None)</code> |
| <pre>Gets details of the specified evaluation result. |
| |
| Args: |
| name: string, Required. The resource name of the evaluation result to retrieve. (required) |
| x__xgafv: string, V1 error format. |
| Allowed values |
| 1 - v1 error format |
| 2 - v2 error format |
| |
| Returns: |
| An object of the form: |
| |
| { # An evaluation result represents the output of running an Evaluation. |
| "appVersion": "A String", # Output only. The app version used to generate the conversation that resulted in this result. Format: `projects/{project}/locations/{location}/apps/{app}/versions/{version}` |
| "appVersionDisplayName": "A String", # Output only. The display name of the `app_version` that the evaluation ran against. |
| "changelog": "A String", # Output only. The changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft. |
| "changelogCreateTime": "A String", # Output only. The create time of the changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft. |
| "config": { # EvaluationConfig configures settings for running the evaluation. # Output only. The configuration used in the evaluation run that resulted in this result. |
| "evaluationChannel": "A String", # Optional. The channel to evaluate. |
| "inputAudioConfig": { # InputAudioConfig configures how the CES agent should interpret the incoming audio data. # Optional. Configuration for processing the input audio. |
| "audioEncoding": "A String", # Required. The encoding of the input audio data. |
| "noiseSuppressionLevel": "A String", # Optional. Whether to enable noise suppression on the input audio. Available values are "low", "moderate", "high", "very_high". |
| "sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the input audio data. |
| }, |
| "outputAudioConfig": { # OutputAudioConfig configures how the CES agent should synthesize outgoing audio responses. # Optional. Configuration for generating the output audio. |
| "audioEncoding": "A String", # Required. The encoding of the output audio data. |
| "sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the output audio data. |
| }, |
| "toolCallBehaviour": "A String", # Optional. Specifies whether the evaluation should use real tool calls or fake tools. |
| }, |
| "createTime": "A String", # Output only. Timestamp when the evaluation result was created. |
| "displayName": "A String", # Required. Display name of the Evaluation Result. Unique within an Evaluation. By default, it has the following format: " result - ". |
| "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Deprecated: Use `error_info` instead. Errors encountered during execution. |
| "code": 42, # The status code, which should be an enum value of google.rpc.Code. |
| "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use. |
| { |
| "a_key": "", # Properties of the object. Contains field @type with type URL. |
| }, |
| ], |
| "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client. |
| }, |
| "errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Error information for the evaluation result. |
| "errorMessage": "A String", # Output only. The error message. |
| "errorType": "A String", # Output only. The type of error. |
| "sessionId": "A String", # Output only. The session ID for the conversation that caused the error. |
| }, |
| "evaluationMetricsThresholds": { # Threshold settings for metrics in an Evaluation. # Output only. The evaluation thresholds for the result. |
| "goldenEvaluationMetricsThresholds": { # Settings for golden evaluations. # Optional. The golden evaluation metrics thresholds. |
| "expectationLevelMetricsThresholds": { # Expectation level metrics thresholds. # Optional. The expectation level metrics thresholds. |
| "toolInvocationParameterCorrectnessThreshold": 3.14, # Optional. The success threshold for individual tool invocation parameter correctness. Must be a float between 0 and 1. Default is 1.0. |
| }, |
| "toolMatchingSettings": { # Settings for matching tool calls. # Optional. The tool matching settings. An extra tool call is a tool call that is present in the execution but does not match any tool call in the golden expectation. |
| "extraToolCallBehavior": "A String", # Optional. Behavior for extra tool calls. Defaults to FAIL. |
| }, |
| "turnLevelMetricsThresholds": { # Turn level metrics thresholds. # Optional. The turn level metrics thresholds. |
| "overallToolInvocationCorrectnessThreshold": 3.14, # Optional. The success threshold for overall tool invocation correctness. Must be a float between 0 and 1. Default is 1.0. |
| "semanticSimilarityChannel": "A String", # Optional. The semantic similarity channel to use for evaluation. |
| "semanticSimilaritySuccessThreshold": 42, # Optional. The success threshold for semantic similarity. Must be an integer between 0 and 4. Default is >= 3. |
| }, |
| }, |
| "goldenHallucinationMetricBehavior": "A String", # Optional. The hallucination metric behavior for golden evaluations. |
| "hallucinationMetricBehavior": "A String", # Optional. Deprecated: Use `golden_hallucination_metric_behavior` instead. The hallucination metric behavior is currently used for golden evaluations. |
| "scenarioHallucinationMetricBehavior": "A String", # Optional. The hallucination metric behavior for scenario evaluations. |
| }, |
| "evaluationRun": "A String", # Output only. The evaluation run that produced this result. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationRuns/{evaluationRun}` |
| "evaluationStatus": "A String", # Output only. The outcome of the evaluation. Only populated if execution_state is COMPLETE. |
| "executionState": "A String", # Output only. The state of the evaluation result execution. |
| "goldenResult": { # The result of a golden evaluation. # Output only. The outcome of a golden evaluation. |
| "evaluationExpectationResults": [ # Output only. The results of the evaluation expectations. |
| { # The result of a single evaluation expectation. |
| "evaluationExpectation": "A String", # Output only. The evaluation expectation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationExpectations/{evaluation_expectation}` |
| "explanation": "A String", # Output only. The explanation for the result. |
| "outcome": "A String", # Output only. The outcome of the evaluation expectation. |
| "prompt": "A String", # Output only. The prompt that was used for the evaluation. |
| }, |
| ], |
| "turnReplayResults": [ # Output only. The result of running each turn of the golden conversation. |
| { # The result of running a single turn of the golden conversation. |
| "conversation": "A String", # Output only. The conversation that was generated for this turn. |
| "errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Information about the error that occurred during this turn. |
| "errorMessage": "A String", # Output only. The error message. |
| "errorType": "A String", # Output only. The type of error. |
| "sessionId": "A String", # Output only. The session ID for the conversation that caused the error. |
| }, |
| "expectationOutcome": [ # Output only. The outcome of each expectation. |
| { # Specifies the expectation and the result of that expectation. |
| "expectation": { # Represents a single, checkable requirement. # Output only. The expectation that was evaluated. |
| "agentResponse": { # A message within a conversation. # Optional. Check that the agent responded with the correct response. The role "agent" is implied. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Check that the agent transferred the conversation to a different agent. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "mockToolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. The tool response to mock, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "note": "A String", # Optional. A note for this requirement, useful in reporting when specific checks fail. E.g., "Check_Payment_Tool_Called". |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Check that a specific tool was called with the parameters. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Check that a specific tool had the expected response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "updatedVariables": { # Optional. Check that the agent updated the session variables to the expected values. Used to also capture agent variable updates for golden evals. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| "observedAgentResponse": { # A message within a conversation. # Output only. The result of the agent response expectation. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "observedAgentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Output only. The result of the agent transfer expectation. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "observedToolCall": { # Request for the client or the agent to execute the specified tool. # Output only. The result of the tool call expectation. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "observedToolResponse": { # The execution result of a specific tool from the client or the agent. # Output only. The result of the tool response expectation. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "outcome": "A String", # Output only. The outcome of the expectation. |
| "semanticSimilarityResult": { # The result of the semantic similarity check. # Output only. The result of the semantic similarity check. |
| "explanation": "A String", # Output only. The explanation for the semantic similarity score. |
| "label": "A String", # Output only. The label associated with each score. Score 4: Fully Consistent Score 3: Mostly Consistent Score 2: Partially Consistent (Minor Omissions) Score 1: Largely Inconsistent (Major Omissions) Score 0: Completely Inconsistent / Contradictory |
| "outcome": "A String", # Output only. The outcome of the semantic similarity check. This is determined by comparing the score to the semantic_similarity_success_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "score": 42, # Output only. The semantic similarity score. Can be 0, 1, 2, 3, or 4. |
| }, |
| "toolInvocationResult": { # The result of the tool invocation check. # Output only. The result of the tool invocation check. |
| "explanation": "A String", # Output only. A free text explanation for the tool invocation result. |
| "outcome": "A String", # Output only. The outcome of the tool invocation check. This is determined by comparing the parameter_correctness_score to the threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "parameterCorrectnessScore": 3.14, # Output only. The tool invocation parameter correctness score. This indicates the percent of parameters from the expected tool call that were also present in the actual tool call. |
| }, |
| }, |
| ], |
| "hallucinationResult": { # The result of the hallucination check for a single turn. # Output only. The result of the hallucination check. |
| "explanation": "A String", # Output only. The explanation for the hallucination score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Justified Score 0: Not Justified Score -1: No Claim To Assess |
| "score": 42, # Output only. The hallucination score. Can be -1, 0, 1. |
| }, |
| "overallToolInvocationResult": { # The result of the overall tool invocation check. # Output only. The result of the overall tool invocation check. |
| "outcome": "A String", # Output only. The outcome of the tool invocation check. This is determined by comparing the tool_invocation_score to the overall_tool_invocation_correctness_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "toolInvocationScore": 3.14, # The overall tool invocation score for this turn. This indicates the overall percent of tools from the expected turn that were actually invoked. |
| }, |
| "semanticSimilarityResult": { # The result of the semantic similarity check. # Output only. The result of the semantic similarity check. |
| "explanation": "A String", # Output only. The explanation for the semantic similarity score. |
| "label": "A String", # Output only. The label associated with each score. Score 4: Fully Consistent Score 3: Mostly Consistent Score 2: Partially Consistent (Minor Omissions) Score 1: Largely Inconsistent (Major Omissions) Score 0: Completely Inconsistent / Contradictory |
| "outcome": "A String", # Output only. The outcome of the semantic similarity check. This is determined by comparing the score to the semantic_similarity_success_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "score": 42, # Output only. The semantic similarity score. Can be 0, 1, 2, 3, or 4. |
| }, |
| "spanLatencies": [ # Output only. The latency of spans in the turn. |
| { # The latency of a span execution. |
| "callback": "A String", # Output only. The name of the user callback span. |
| "displayName": "A String", # Output only. The display name of the span. Applicable to tool and guardrail spans. |
| "endTime": "A String", # Output only. The end time of span. |
| "executionLatency": "A String", # Output only. The latency of span. |
| "model": "A String", # Output only. The name of the LLM span. |
| "resource": "A String", # Output only. The resource name of the guardrail or tool spans. |
| "startTime": "A String", # Output only. The start time of span. |
| "toolset": { # A tool that is created from a toolset. # Output only. The toolset tool identifier. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| "type": "A String", # Output only. The type of span. |
| }, |
| ], |
| "toolCallLatencies": [ # Output only. The latency of each tool call in the turn. |
| { # The latency of a tool call execution. |
| "displayName": "A String", # Output only. The display name of the tool. |
| "endTime": "A String", # Output only. The end time of the tool call execution. |
| "executionLatency": "A String", # Output only. The latency of the tool call execution. |
| "startTime": "A String", # Output only. The start time of the tool call execution. |
| "tool": "A String", # Output only. The name of the tool that got executed. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`. |
| }, |
| ], |
| "toolInvocationScore": 3.14, # Output only. Deprecated. Use OverallToolInvocationResult instead. |
| "toolOrderedInvocationScore": 3.14, # Output only. The overall tool ordered invocation score for this turn. This indicates the overall percent of tools from the expected turn that were actually invoked in the expected order. |
| "turnLatency": "A String", # Output only. Duration of the turn. |
| }, |
| ], |
| }, |
| "goldenRunMethod": "A String", # Output only. The method used to run the golden evaluation. |
| "initiatedBy": "A String", # Output only. The user who initiated the evaluation run that resulted in this result. |
| "name": "A String", # Identifier. The unique identifier of the evaluation result. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}/results/{result}` |
| "persona": { # A persona represents an end user in an evaluation. # Output only. The persona used to generate the conversation for the evaluation result. |
| "description": "A String", # Optional. The description of the persona. |
| "displayName": "A String", # Required. The display name of the persona. Unique within an app. |
| "name": "A String", # Required. The unique identifier of the persona. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationPersonas/{evaluationPersona}` |
| "personality": "A String", # Required. An instruction for the agent on how to behave in the evaluation. |
| "speechConfig": { # Configuration for Text-to-Speech generation. # Optional. Configuration for how the persona sounds (TTS settings). |
| "environment": "A String", # Optional. The simulated audio environment. |
| "speakingRate": 3.14, # Optional. The speaking rate. 1.0 is normal. Lower is slower (e.g., 0.8), higher is faster (e.g., 1.5). Useful for testing how the agent handles fast talkers. |
| "voiceId": "A String", # Optional. The specific voice identifier/accent to use. Example: "en-US-Wavenet-D" or "en-GB-Standard-A" |
| }, |
| }, |
| "scenarioResult": { # The outcome of a scenario evaluation. # Output only. The outcome of a scenario evaluation. |
| "allExpectationsSatisfied": True or False, # Output only. Whether all expectations were satisfied for this turn. |
| "conversation": "A String", # Output only. The conversation that was generated in the scenario. |
| "evaluationExpectationResults": [ # Output only. The results of the evaluation expectations. |
| { # The result of a single evaluation expectation. |
| "evaluationExpectation": "A String", # Output only. The evaluation expectation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationExpectations/{evaluation_expectation}` |
| "explanation": "A String", # Output only. The explanation for the result. |
| "outcome": "A String", # Output only. The outcome of the evaluation expectation. |
| "prompt": "A String", # Output only. The prompt that was used for the evaluation. |
| }, |
| ], |
| "expectationOutcomes": [ # Output only. The outcome of each expectation. |
| { # The outcome of a scenario expectation. |
| "expectation": { # The expectation to evaluate the conversation produced by the simulation. # Output only. The expectation that was evaluated. |
| "agentResponse": { # A message within a conversation. # Optional. The agent response to be evaluated. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "toolExpectation": { # The tool call and response pair to be evaluated. # Optional. The tool call and response pair to be evaluated. |
| "expectedToolCall": { # Request for the client or the agent to execute the specified tool. # Required. The expected tool call, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "mockToolResponse": { # The execution result of a specific tool from the client or the agent. # Required. The tool response to mock, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| }, |
| }, |
| "observedAgentResponse": { # A message within a conversation. # Output only. The observed agent response. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "observedToolCall": { # The observed tool call and response. # Output only. The observed tool call. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Output only. The observed tool call. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Output only. The observed tool response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| }, |
| "outcome": "A String", # Output only. The outcome of the ScenarioExpectation. |
| }, |
| ], |
| "hallucinationResult": [ # Output only. The result of the hallucination check. There will be one hallucination result for each turn in the conversation. |
| { # The result of the hallucination check for a single turn. |
| "explanation": "A String", # Output only. The explanation for the hallucination score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Justified Score 0: Not Justified Score -1: No Claim To Assess |
| "score": 42, # Output only. The hallucination score. Can be -1, 0, 1. |
| }, |
| ], |
| "rubricOutcomes": [ # Output only. The outcome of the rubric. |
| { # The outcome of the evaluation against the rubric. |
| "rubric": "A String", # Output only. The rubric that was used to evaluate the conversation. |
| "score": 3.14, # Output only. The score of the conversation against the rubric. |
| "scoreExplanation": "A String", # Output only. The rater's response to the rubric. |
| }, |
| ], |
| "spanLatencies": [ # Output only. The latency of spans in the conversation. |
| { # The latency of a span execution. |
| "callback": "A String", # Output only. The name of the user callback span. |
| "displayName": "A String", # Output only. The display name of the span. Applicable to tool and guardrail spans. |
| "endTime": "A String", # Output only. The end time of span. |
| "executionLatency": "A String", # Output only. The latency of span. |
| "model": "A String", # Output only. The name of the LLM span. |
| "resource": "A String", # Output only. The resource name of the guardrail or tool spans. |
| "startTime": "A String", # Output only. The start time of span. |
| "toolset": { # A tool that is created from a toolset. # Output only. The toolset tool identifier. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| "type": "A String", # Output only. The type of span. |
| }, |
| ], |
| "task": "A String", # Output only. The task that was used when running the scenario for this result. |
| "taskCompleted": True or False, # Output only. Whether the task was completed for this turn. This is a composite of all expectations satisfied, no hallucinations, and user goal satisfaction. |
| "taskCompletionResult": { # The result of the task completion check for the conversation. # Output only. The result of the task completion check. |
| "explanation": "A String", # Output only. The explanation for the task completion score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Task Completed Score 0: Task Not Completed Score -1: User Goal Undefined |
| "score": 42, # Output only. The task completion score. Can be -1, 0, 1 |
| }, |
| "toolCallLatencies": [ # Output only. The latency of each tool call execution in the conversation. |
| { # The latency of a tool call execution. |
| "displayName": "A String", # Output only. The display name of the tool. |
| "endTime": "A String", # Output only. The end time of the tool call execution. |
| "executionLatency": "A String", # Output only. The latency of the tool call execution. |
| "startTime": "A String", # Output only. The start time of the tool call execution. |
| "tool": "A String", # Output only. The name of the tool that got executed. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`. |
| }, |
| ], |
| "userFacts": [ # Output only. The user facts that were used by the scenario for this result. |
| { # Facts about the user as a key value pair. |
| "name": "A String", # Required. The name of the user fact. |
| "value": "A String", # Required. The value of the user fact. |
| }, |
| ], |
| "userGoalSatisfactionResult": { # The result of a user goal satisfaction check for a conversation. # Output only. The result of the user goal satisfaction check. |
| "explanation": "A String", # Output only. The explanation for the user task satisfaction score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: User Task Satisfied Score 0: User Task Not Satisfied Score -1: User Task Unspecified |
| "score": 42, # Output only. The user task satisfaction score. Can be -1, 0, 1. |
| }, |
| }, |
| }</pre> |
| </div> |
| |
| <div class="method"> |
| <code class="details" id="list">list(parent, filter=None, orderBy=None, pageSize=None, pageToken=None, x__xgafv=None)</code> |
| <pre>Lists all evaluation results for a given evaluation. |
| |
| Args: |
| parent: string, Required. The resource name of the evaluation to list evaluation results from. To filter by evaluation run, use `-` as the evaluation ID and specify the evaluation run ID in the filter. For example: `projects/{project}/locations/{location}/apps/{app}/evaluations/-` (required) |
| filter: string, Optional. Filter to be applied when listing the evaluation results. See https://google.aip.dev/160 for more details. |
| orderBy: string, Optional. Field to sort by. Only "name" and "create_time", and "update_time" are supported. Time fields are ordered in descending order, and the name field is ordered in ascending order. If not included, "update_time" will be the default. See https://google.aip.dev/132#ordering for more details. |
| pageSize: integer, Optional. Requested page size. Server may return fewer items than requested. If unspecified, server will pick an appropriate default. |
| pageToken: string, Optional. The next_page_token value returned from a previous list EvaluationService.ListEvaluationResults call. |
| x__xgafv: string, V1 error format. |
| Allowed values |
| 1 - v1 error format |
| 2 - v2 error format |
| |
| Returns: |
| An object of the form: |
| |
| { # Response message for EvaluationService.ListEvaluationResults. |
| "evaluationResults": [ # The list of evaluation results. |
| { # An evaluation result represents the output of running an Evaluation. |
| "appVersion": "A String", # Output only. The app version used to generate the conversation that resulted in this result. Format: `projects/{project}/locations/{location}/apps/{app}/versions/{version}` |
| "appVersionDisplayName": "A String", # Output only. The display name of the `app_version` that the evaluation ran against. |
| "changelog": "A String", # Output only. The changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft. |
| "changelogCreateTime": "A String", # Output only. The create time of the changelog of the app version that the evaluation ran against. This is populated if user runs evaluation on latest/draft. |
| "config": { # EvaluationConfig configures settings for running the evaluation. # Output only. The configuration used in the evaluation run that resulted in this result. |
| "evaluationChannel": "A String", # Optional. The channel to evaluate. |
| "inputAudioConfig": { # InputAudioConfig configures how the CES agent should interpret the incoming audio data. # Optional. Configuration for processing the input audio. |
| "audioEncoding": "A String", # Required. The encoding of the input audio data. |
| "noiseSuppressionLevel": "A String", # Optional. Whether to enable noise suppression on the input audio. Available values are "low", "moderate", "high", "very_high". |
| "sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the input audio data. |
| }, |
| "outputAudioConfig": { # OutputAudioConfig configures how the CES agent should synthesize outgoing audio responses. # Optional. Configuration for generating the output audio. |
| "audioEncoding": "A String", # Required. The encoding of the output audio data. |
| "sampleRateHertz": 42, # Required. The sample rate (in Hertz) of the output audio data. |
| }, |
| "toolCallBehaviour": "A String", # Optional. Specifies whether the evaluation should use real tool calls or fake tools. |
| }, |
| "createTime": "A String", # Output only. Timestamp when the evaluation result was created. |
| "displayName": "A String", # Required. Display name of the Evaluation Result. Unique within an Evaluation. By default, it has the following format: " result - ". |
| "error": { # The `Status` type defines a logical error model that is suitable for different programming environments, including REST APIs and RPC APIs. It is used by [gRPC](https://github.com/grpc). Each `Status` message contains three pieces of data: error code, error message, and error details. You can find out more about this error model and how to work with it in the [API Design Guide](https://cloud.google.com/apis/design/errors). # Output only. Deprecated: Use `error_info` instead. Errors encountered during execution. |
| "code": 42, # The status code, which should be an enum value of google.rpc.Code. |
| "details": [ # A list of messages that carry the error details. There is a common set of message types for APIs to use. |
| { |
| "a_key": "", # Properties of the object. Contains field @type with type URL. |
| }, |
| ], |
| "message": "A String", # A developer-facing error message, which should be in English. Any user-facing error message should be localized and sent in the google.rpc.Status.details field, or localized by the client. |
| }, |
| "errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Error information for the evaluation result. |
| "errorMessage": "A String", # Output only. The error message. |
| "errorType": "A String", # Output only. The type of error. |
| "sessionId": "A String", # Output only. The session ID for the conversation that caused the error. |
| }, |
| "evaluationMetricsThresholds": { # Threshold settings for metrics in an Evaluation. # Output only. The evaluation thresholds for the result. |
| "goldenEvaluationMetricsThresholds": { # Settings for golden evaluations. # Optional. The golden evaluation metrics thresholds. |
| "expectationLevelMetricsThresholds": { # Expectation level metrics thresholds. # Optional. The expectation level metrics thresholds. |
| "toolInvocationParameterCorrectnessThreshold": 3.14, # Optional. The success threshold for individual tool invocation parameter correctness. Must be a float between 0 and 1. Default is 1.0. |
| }, |
| "toolMatchingSettings": { # Settings for matching tool calls. # Optional. The tool matching settings. An extra tool call is a tool call that is present in the execution but does not match any tool call in the golden expectation. |
| "extraToolCallBehavior": "A String", # Optional. Behavior for extra tool calls. Defaults to FAIL. |
| }, |
| "turnLevelMetricsThresholds": { # Turn level metrics thresholds. # Optional. The turn level metrics thresholds. |
| "overallToolInvocationCorrectnessThreshold": 3.14, # Optional. The success threshold for overall tool invocation correctness. Must be a float between 0 and 1. Default is 1.0. |
| "semanticSimilarityChannel": "A String", # Optional. The semantic similarity channel to use for evaluation. |
| "semanticSimilaritySuccessThreshold": 42, # Optional. The success threshold for semantic similarity. Must be an integer between 0 and 4. Default is >= 3. |
| }, |
| }, |
| "goldenHallucinationMetricBehavior": "A String", # Optional. The hallucination metric behavior for golden evaluations. |
| "hallucinationMetricBehavior": "A String", # Optional. Deprecated: Use `golden_hallucination_metric_behavior` instead. The hallucination metric behavior is currently used for golden evaluations. |
| "scenarioHallucinationMetricBehavior": "A String", # Optional. The hallucination metric behavior for scenario evaluations. |
| }, |
| "evaluationRun": "A String", # Output only. The evaluation run that produced this result. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationRuns/{evaluationRun}` |
| "evaluationStatus": "A String", # Output only. The outcome of the evaluation. Only populated if execution_state is COMPLETE. |
| "executionState": "A String", # Output only. The state of the evaluation result execution. |
| "goldenResult": { # The result of a golden evaluation. # Output only. The outcome of a golden evaluation. |
| "evaluationExpectationResults": [ # Output only. The results of the evaluation expectations. |
| { # The result of a single evaluation expectation. |
| "evaluationExpectation": "A String", # Output only. The evaluation expectation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationExpectations/{evaluation_expectation}` |
| "explanation": "A String", # Output only. The explanation for the result. |
| "outcome": "A String", # Output only. The outcome of the evaluation expectation. |
| "prompt": "A String", # Output only. The prompt that was used for the evaluation. |
| }, |
| ], |
| "turnReplayResults": [ # Output only. The result of running each turn of the golden conversation. |
| { # The result of running a single turn of the golden conversation. |
| "conversation": "A String", # Output only. The conversation that was generated for this turn. |
| "errorInfo": { # Information about an error encountered during an evaluation execution. # Output only. Information about the error that occurred during this turn. |
| "errorMessage": "A String", # Output only. The error message. |
| "errorType": "A String", # Output only. The type of error. |
| "sessionId": "A String", # Output only. The session ID for the conversation that caused the error. |
| }, |
| "expectationOutcome": [ # Output only. The outcome of each expectation. |
| { # Specifies the expectation and the result of that expectation. |
| "expectation": { # Represents a single, checkable requirement. # Output only. The expectation that was evaluated. |
| "agentResponse": { # A message within a conversation. # Optional. Check that the agent responded with the correct response. The role "agent" is implied. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Check that the agent transferred the conversation to a different agent. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "mockToolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. The tool response to mock, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "note": "A String", # Optional. A note for this requirement, useful in reporting when specific checks fail. E.g., "Check_Payment_Tool_Called". |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Check that a specific tool was called with the parameters. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Check that a specific tool had the expected response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "updatedVariables": { # Optional. Check that the agent updated the session variables to the expected values. Used to also capture agent variable updates for golden evals. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| "observedAgentResponse": { # A message within a conversation. # Output only. The result of the agent response expectation. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "observedAgentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Output only. The result of the agent transfer expectation. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "observedToolCall": { # Request for the client or the agent to execute the specified tool. # Output only. The result of the tool call expectation. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "observedToolResponse": { # The execution result of a specific tool from the client or the agent. # Output only. The result of the tool response expectation. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "outcome": "A String", # Output only. The outcome of the expectation. |
| "semanticSimilarityResult": { # The result of the semantic similarity check. # Output only. The result of the semantic similarity check. |
| "explanation": "A String", # Output only. The explanation for the semantic similarity score. |
| "label": "A String", # Output only. The label associated with each score. Score 4: Fully Consistent Score 3: Mostly Consistent Score 2: Partially Consistent (Minor Omissions) Score 1: Largely Inconsistent (Major Omissions) Score 0: Completely Inconsistent / Contradictory |
| "outcome": "A String", # Output only. The outcome of the semantic similarity check. This is determined by comparing the score to the semantic_similarity_success_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "score": 42, # Output only. The semantic similarity score. Can be 0, 1, 2, 3, or 4. |
| }, |
| "toolInvocationResult": { # The result of the tool invocation check. # Output only. The result of the tool invocation check. |
| "explanation": "A String", # Output only. A free text explanation for the tool invocation result. |
| "outcome": "A String", # Output only. The outcome of the tool invocation check. This is determined by comparing the parameter_correctness_score to the threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "parameterCorrectnessScore": 3.14, # Output only. The tool invocation parameter correctness score. This indicates the percent of parameters from the expected tool call that were also present in the actual tool call. |
| }, |
| }, |
| ], |
| "hallucinationResult": { # The result of the hallucination check for a single turn. # Output only. The result of the hallucination check. |
| "explanation": "A String", # Output only. The explanation for the hallucination score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Justified Score 0: Not Justified Score -1: No Claim To Assess |
| "score": 42, # Output only. The hallucination score. Can be -1, 0, 1. |
| }, |
| "overallToolInvocationResult": { # The result of the overall tool invocation check. # Output only. The result of the overall tool invocation check. |
| "outcome": "A String", # Output only. The outcome of the tool invocation check. This is determined by comparing the tool_invocation_score to the overall_tool_invocation_correctness_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "toolInvocationScore": 3.14, # The overall tool invocation score for this turn. This indicates the overall percent of tools from the expected turn that were actually invoked. |
| }, |
| "semanticSimilarityResult": { # The result of the semantic similarity check. # Output only. The result of the semantic similarity check. |
| "explanation": "A String", # Output only. The explanation for the semantic similarity score. |
| "label": "A String", # Output only. The label associated with each score. Score 4: Fully Consistent Score 3: Mostly Consistent Score 2: Partially Consistent (Minor Omissions) Score 1: Largely Inconsistent (Major Omissions) Score 0: Completely Inconsistent / Contradictory |
| "outcome": "A String", # Output only. The outcome of the semantic similarity check. This is determined by comparing the score to the semantic_similarity_success_threshold. If the score is equal to or above the threshold, the outcome will be PASS. Otherwise, the outcome will be FAIL. |
| "score": 42, # Output only. The semantic similarity score. Can be 0, 1, 2, 3, or 4. |
| }, |
| "spanLatencies": [ # Output only. The latency of spans in the turn. |
| { # The latency of a span execution. |
| "callback": "A String", # Output only. The name of the user callback span. |
| "displayName": "A String", # Output only. The display name of the span. Applicable to tool and guardrail spans. |
| "endTime": "A String", # Output only. The end time of span. |
| "executionLatency": "A String", # Output only. The latency of span. |
| "model": "A String", # Output only. The name of the LLM span. |
| "resource": "A String", # Output only. The resource name of the guardrail or tool spans. |
| "startTime": "A String", # Output only. The start time of span. |
| "toolset": { # A tool that is created from a toolset. # Output only. The toolset tool identifier. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| "type": "A String", # Output only. The type of span. |
| }, |
| ], |
| "toolCallLatencies": [ # Output only. The latency of each tool call in the turn. |
| { # The latency of a tool call execution. |
| "displayName": "A String", # Output only. The display name of the tool. |
| "endTime": "A String", # Output only. The end time of the tool call execution. |
| "executionLatency": "A String", # Output only. The latency of the tool call execution. |
| "startTime": "A String", # Output only. The start time of the tool call execution. |
| "tool": "A String", # Output only. The name of the tool that got executed. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`. |
| }, |
| ], |
| "toolInvocationScore": 3.14, # Output only. Deprecated. Use OverallToolInvocationResult instead. |
| "toolOrderedInvocationScore": 3.14, # Output only. The overall tool ordered invocation score for this turn. This indicates the overall percent of tools from the expected turn that were actually invoked in the expected order. |
| "turnLatency": "A String", # Output only. Duration of the turn. |
| }, |
| ], |
| }, |
| "goldenRunMethod": "A String", # Output only. The method used to run the golden evaluation. |
| "initiatedBy": "A String", # Output only. The user who initiated the evaluation run that resulted in this result. |
| "name": "A String", # Identifier. The unique identifier of the evaluation result. Format: `projects/{project}/locations/{location}/apps/{app}/evaluations/{evaluation}/results/{result}` |
| "persona": { # A persona represents an end user in an evaluation. # Output only. The persona used to generate the conversation for the evaluation result. |
| "description": "A String", # Optional. The description of the persona. |
| "displayName": "A String", # Required. The display name of the persona. Unique within an app. |
| "name": "A String", # Required. The unique identifier of the persona. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationPersonas/{evaluationPersona}` |
| "personality": "A String", # Required. An instruction for the agent on how to behave in the evaluation. |
| "speechConfig": { # Configuration for Text-to-Speech generation. # Optional. Configuration for how the persona sounds (TTS settings). |
| "environment": "A String", # Optional. The simulated audio environment. |
| "speakingRate": 3.14, # Optional. The speaking rate. 1.0 is normal. Lower is slower (e.g., 0.8), higher is faster (e.g., 1.5). Useful for testing how the agent handles fast talkers. |
| "voiceId": "A String", # Optional. The specific voice identifier/accent to use. Example: "en-US-Wavenet-D" or "en-GB-Standard-A" |
| }, |
| }, |
| "scenarioResult": { # The outcome of a scenario evaluation. # Output only. The outcome of a scenario evaluation. |
| "allExpectationsSatisfied": True or False, # Output only. Whether all expectations were satisfied for this turn. |
| "conversation": "A String", # Output only. The conversation that was generated in the scenario. |
| "evaluationExpectationResults": [ # Output only. The results of the evaluation expectations. |
| { # The result of a single evaluation expectation. |
| "evaluationExpectation": "A String", # Output only. The evaluation expectation. Format: `projects/{project}/locations/{location}/apps/{app}/evaluationExpectations/{evaluation_expectation}` |
| "explanation": "A String", # Output only. The explanation for the result. |
| "outcome": "A String", # Output only. The outcome of the evaluation expectation. |
| "prompt": "A String", # Output only. The prompt that was used for the evaluation. |
| }, |
| ], |
| "expectationOutcomes": [ # Output only. The outcome of each expectation. |
| { # The outcome of a scenario expectation. |
| "expectation": { # The expectation to evaluate the conversation produced by the simulation. # Output only. The expectation that was evaluated. |
| "agentResponse": { # A message within a conversation. # Optional. The agent response to be evaluated. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "toolExpectation": { # The tool call and response pair to be evaluated. # Optional. The tool call and response pair to be evaluated. |
| "expectedToolCall": { # Request for the client or the agent to execute the specified tool. # Required. The expected tool call, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "mockToolResponse": { # The execution result of a specific tool from the client or the agent. # Required. The tool response to mock, with the parameters of interest specified. Any parameters not specified will be hallucinated by the LLM. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| }, |
| }, |
| "observedAgentResponse": { # A message within a conversation. # Output only. The observed agent response. |
| "chunks": [ # Optional. Content of the message as a series of chunks. |
| { # A chunk of content within a message. |
| "agentTransfer": { # Represents an event indicating the transfer of a conversation to a different agent. # Optional. Agent transfer event. |
| "displayName": "A String", # Output only. Display name of the agent. |
| "targetAgent": "A String", # Required. The agent to which the conversation is being transferred. The agent will handle the conversation from this point forward. Format: `projects/{project}/locations/{location}/apps/{app}/agents/{agent}` |
| }, |
| "blob": { # Represents a blob input or output in the conversation. # Optional. Blob data. |
| "data": "A String", # Required. Raw bytes of the blob. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. |
| }, |
| "defaultVariables": { # A struct represents default variables at the start of the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| "image": { # Represents an image input or output in the conversation. # Optional. Image data. |
| "data": "A String", # Required. Raw bytes of the image. |
| "mimeType": "A String", # Required. The IANA standard MIME type of the source data. Supported image types includes: * image/png * image/jpeg * image/webp |
| }, |
| "payload": { # Optional. Custom payload data. |
| "a_key": "", # Properties of the object. |
| }, |
| "text": "A String", # Optional. Text data. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Optional. Tool execution request. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Optional. Tool execution response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "transcript": "A String", # Optional. Transcript associated with the audio. |
| "updatedVariables": { # A struct represents variables that were updated in the conversation, keyed by variable names. |
| "a_key": "", # Properties of the object. |
| }, |
| }, |
| ], |
| "eventTime": "A String", # Optional. Timestamp when the message was sent or received. Should not be used if the message is part of an example. |
| "role": "A String", # Optional. The role within the conversation, e.g., user, agent. |
| }, |
| "observedToolCall": { # The observed tool call and response. # Output only. The observed tool call. |
| "toolCall": { # Request for the client or the agent to execute the specified tool. # Output only. The observed tool call. |
| "args": { # Optional. The input parameters and values for the tool in JSON object format. |
| "a_key": "", # Properties of the object. |
| }, |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The unique identifier of the tool call. If populated, the client should return the execution result with the matching ID in ToolResponse. |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool to execute. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| "toolResponse": { # The execution result of a specific tool from the client or the agent. # Output only. The observed tool response. |
| "displayName": "A String", # Output only. Display name of the tool. |
| "id": "A String", # Optional. The matching ID of the tool call the response is for. |
| "response": { # Required. The tool execution result in JSON object format. Use "output" key to specify tool response and "error" key to specify error details (if any). If "output" and "error" keys are not specified, then whole "response" is treated as tool execution result. |
| "a_key": "", # Properties of the object. |
| }, |
| "tool": "A String", # Optional. The name of the tool to execute. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}` |
| "toolsetTool": { # A tool that is created from a toolset. # Optional. The toolset tool that got executed. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| }, |
| }, |
| "outcome": "A String", # Output only. The outcome of the ScenarioExpectation. |
| }, |
| ], |
| "hallucinationResult": [ # Output only. The result of the hallucination check. There will be one hallucination result for each turn in the conversation. |
| { # The result of the hallucination check for a single turn. |
| "explanation": "A String", # Output only. The explanation for the hallucination score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Justified Score 0: Not Justified Score -1: No Claim To Assess |
| "score": 42, # Output only. The hallucination score. Can be -1, 0, 1. |
| }, |
| ], |
| "rubricOutcomes": [ # Output only. The outcome of the rubric. |
| { # The outcome of the evaluation against the rubric. |
| "rubric": "A String", # Output only. The rubric that was used to evaluate the conversation. |
| "score": 3.14, # Output only. The score of the conversation against the rubric. |
| "scoreExplanation": "A String", # Output only. The rater's response to the rubric. |
| }, |
| ], |
| "spanLatencies": [ # Output only. The latency of spans in the conversation. |
| { # The latency of a span execution. |
| "callback": "A String", # Output only. The name of the user callback span. |
| "displayName": "A String", # Output only. The display name of the span. Applicable to tool and guardrail spans. |
| "endTime": "A String", # Output only. The end time of span. |
| "executionLatency": "A String", # Output only. The latency of span. |
| "model": "A String", # Output only. The name of the LLM span. |
| "resource": "A String", # Output only. The resource name of the guardrail or tool spans. |
| "startTime": "A String", # Output only. The start time of span. |
| "toolset": { # A tool that is created from a toolset. # Output only. The toolset tool identifier. |
| "toolId": "A String", # Optional. The tool ID to filter the tools to retrieve the schema for. |
| "toolset": "A String", # Required. The resource name of the Toolset from which this tool is derived. Format: `projects/{project}/locations/{location}/apps/{app}/toolsets/{toolset}` |
| }, |
| "type": "A String", # Output only. The type of span. |
| }, |
| ], |
| "task": "A String", # Output only. The task that was used when running the scenario for this result. |
| "taskCompleted": True or False, # Output only. Whether the task was completed for this turn. This is a composite of all expectations satisfied, no hallucinations, and user goal satisfaction. |
| "taskCompletionResult": { # The result of the task completion check for the conversation. # Output only. The result of the task completion check. |
| "explanation": "A String", # Output only. The explanation for the task completion score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: Task Completed Score 0: Task Not Completed Score -1: User Goal Undefined |
| "score": 42, # Output only. The task completion score. Can be -1, 0, 1 |
| }, |
| "toolCallLatencies": [ # Output only. The latency of each tool call execution in the conversation. |
| { # The latency of a tool call execution. |
| "displayName": "A String", # Output only. The display name of the tool. |
| "endTime": "A String", # Output only. The end time of the tool call execution. |
| "executionLatency": "A String", # Output only. The latency of the tool call execution. |
| "startTime": "A String", # Output only. The start time of the tool call execution. |
| "tool": "A String", # Output only. The name of the tool that got executed. Format: `projects/{project}/locations/{location}/apps/{app}/tools/{tool}`. |
| }, |
| ], |
| "userFacts": [ # Output only. The user facts that were used by the scenario for this result. |
| { # Facts about the user as a key value pair. |
| "name": "A String", # Required. The name of the user fact. |
| "value": "A String", # Required. The value of the user fact. |
| }, |
| ], |
| "userGoalSatisfactionResult": { # The result of a user goal satisfaction check for a conversation. # Output only. The result of the user goal satisfaction check. |
| "explanation": "A String", # Output only. The explanation for the user task satisfaction score. |
| "label": "A String", # Output only. The label associated with each score. Score 1: User Task Satisfied Score 0: User Task Not Satisfied Score -1: User Task Unspecified |
| "score": 42, # Output only. The user task satisfaction score. Can be -1, 0, 1. |
| }, |
| }, |
| }, |
| ], |
| "nextPageToken": "A String", # A token that can be sent as ListEvaluationResultsRequest.page_token to retrieve the next page. Absence of this field indicates there are no subsequent pages. |
| }</pre> |
| </div> |
| |
| <div class="method"> |
| <code class="details" id="list_next">list_next()</code> |
| <pre>Retrieves the next page of results. |
| |
| Args: |
| previous_request: The request for the previous page. (required) |
| previous_response: The response from the request for the previous page. (required) |
| |
| Returns: |
| A request object that you can call 'execute()' on to request the next |
| page. Returns None if there are no more items in the collection. |
| </pre> |
| </div> |
| |
| </body></html> |
