promptquality.types package

Subpackages

Submodules

promptquality.types.config module

pydantic settings Config

Bases: BaseSettings

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Show JSON schema
{
   "title": "Config",
   "type": "object",
   "properties": {
      "console_url": {
         "format": "uri",
         "maxLength": 2083,
         "minLength": 1,
         "title": "Console Url",
         "type": "string"
      },
      "username": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Username"
      },
      "password": {
         "anyOf": [
            {
               "format": "password",
               "type": "string",
               "writeOnly": true
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Password"
      },
      "api_key": {
         "anyOf": [
            {
               "format": "password",
               "type": "string",
               "writeOnly": true
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Api Key"
      },
      "token": {
         "anyOf": [
            {
               "format": "password",
               "type": "string",
               "writeOnly": true
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Token"
      },
      "current_user": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current User"
      },
      "current_project_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Project Id"
      },
      "current_project_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Project Name"
      },
      "current_run_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Run Id"
      },
      "current_run_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Run Name"
      },
      "current_run_url": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Run Url"
      },
      "current_run_task_type": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Run Task Type"
      },
      "current_template_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Template Id"
      },
      "current_template_name": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Template Name"
      },
      "current_template_version_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Template Version Id"
      },
      "current_template_version": {
         "anyOf": [
            {
               "type": "integer"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Template Version"
      },
      "current_template": {
         "anyOf": [
            {
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Template"
      },
      "current_dataset_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Dataset Id"
      },
      "current_job_id": {
         "anyOf": [
            {
               "format": "uuid4",
               "type": "string"
            },
            {
               "type": "null"
            }
         ],
         "default": null,
         "title": "Current Job Id"
      }
   },
   "required": [
      "console_url"
   ]
}

Config:

  • extra: str = ignore

  • env_prefix: str = GALILEO_

  • validate_assignment: bool = True

Fields:
Validators:
field api_key: Optional[SecretStr] = None
Validated by:
field console_url: Annotated[Url] [Required]
Constraints:

  • max_length = 2083

  • allowed_schemes = [‘http’, ‘https’]

Validated by:
field current_dataset_id: Optional[Annotated[UUID]] = None
Validated by:
field current_job_id: Optional[Annotated[UUID]] = None
Validated by:
field current_project_id: Optional[Annotated[UUID]] = None
Validated by:
field current_project_name: Optional[str] = None
Validated by:
field current_run_id: Optional[Annotated[UUID]] = None
Validated by:
field current_run_name: Optional[str] = None
Validated by:
field current_run_task_type: Optional[int] = None
Validated by:
field current_run_url: Optional[str] = None
Validated by:
field current_template: Optional[str] = None
Validated by:
field current_template_id: Optional[Annotated[UUID]] = None
Validated by:
field current_template_name: Optional[str] = None
Validated by:
field current_template_version: Optional[int] = None
Validated by:
field current_template_version_id: Optional[Annotated[UUID]] = None
Validated by:
field current_user: Optional[str] = None
Validated by:
field password: Optional[SecretStr] = None
Validated by:
field token: Optional[SecretStr] = None
Validated by:
field username: Optional[str] = None
Validated by:
validator http_url  »  console_url
Return type:

str

login()
Return type:

None

logout()
Return type:

None

merge_dataset(dataset)
Return type:

None

merge_job(job_info)
Return type:

None

merge_project(project)
Return type:

None

merge_run(run)
Return type:

None

merge_template(template)
Return type:

None

merge_template_version(template_version)
Return type:

None

classmethod read()
Return type:

Config

serialize_token(value)
Return type:

str

token_login()
Return type:

str

validator validate_api_url  »  all fields
Return type:

Config

write()
Return type:

None

property api_client: ApiClient
property api_url: Url
property config_file: Path
property project_url: str

promptquality.types.custom_scorer module

pydantic model CustomScorer

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field aggregator_fn: Optional[Callable[[List[Union[float, int, bool, str, None]], List[int]], Dict[str, Union[float, int, bool, str, None]]]] = None
field name: str [Required]
Validated by:

  • validate_scorer_name

field scorer_fn: Callable[[PromptRow], Union[float, int, bool, str, None]] [Required]

promptquality.types.pagination module

pydantic model PaginationRequestMixin

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field limit: int = 25
field starting_token: int = 0
pydantic model PaginationResponseMixin

Bases: PaginationRequestMixin, BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field next_starting_token: Optional[int] = None
field paginated: bool = False

promptquality.types.prompt_optimization module

pydantic model PromptOptimizationConfiguration

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field evaluation_criteria: str [Required]
field evaluation_model_alias: str = Models.gpt_35_turbo_16k_0125

The model alias to use for evaluating prompts. The model alias must be a valid alias within Galileo, see promptquality.constants.models.Models for available models. The default value is the GPT-3.5-turbo model with a max tokens of 16k and a temperature of 0.125.

field generation_model_alias: str = Models.gpt_35_turbo_16k_0125

The model alias to use for generating prompts. The model alias must be a valid alias within Galileo, see promptquality.constants.models.Models for available models. The default value is the GPT-3.5-turbo model with a max tokens of 16k and a temperature of 0.125.

field iterations: int = 10

Number of iterations to run the optimization. More iterations will result in a better prompt, but will take longer to run and cost more. Galileo research shows that 10 iterations balances quality and cost to produce optimal results.

Constraints:

  • gt = 0

field max_tokens: int = 4096
field num_train_rows: int = 30

Number of rows randomly sampled from the dataset to use for training the optimized prompt. A larger value will produce longer runtime and incur a higher cost. Galileo research supports that values lower than 30 will not provide an improved prompt.

Constraints:

  • gt = 0

field num_val_rows: int = 100

Number of rows randomly sampled from the dataset to use for validation of the prompt score. A larger value will produce longer runtime and incur a higher cost. Galileo research supports that values lower than 100 will not provide a reliable validation score.

Constraints:

  • gt = 0

field prompt: str [Required]
field task_description: str [Required]
field temperature: float = 1.0
field validation_dataset_id: Optional[Annotated[UUID]] = None

promptquality.types.registered_scorers module

pydantic model RegisteredScorer

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field id: Annotated[UUID] [Required] (alias 'registered_scorer_id')
Constraints:

  • uuid_version = 4

field name: str [Required] (alias 'metric_name')
pydantic model ListRegisteredScorers

Bases: PaginationResponseMixin

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field scorers: List[RegisteredScorer] [Required]

promptquality.types.rows module

pydantic model GetRowsRequest

Bases: PaginationRequestMixin

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field run_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field task_type: int [Required]
params()

Params to be passed to the API request.

These are primarily the pagination parameters and task type.

Returns:

Params to be passed to the API request.

Return type:

Dict[str, Any]

pydantic model Metrics

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field bleu: Optional[float] = None
field context_adherence: Optional[float] = None
field context_adherence_explanation: Optional[str] = None
field context_relevance: Optional[float] = None
field correctness: Optional[float] = None
field correctness_explanation: Optional[str] = None
field latency: Optional[float] = None
field like_dislike: Optional[bool] = None
field pii: Optional[List[str]] [Optional]
field rouge: Optional[float] = None
field toxicity: Optional[float] = None
field uncertainty: Optional[float] = None
pydantic model PromptRow

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field bleu: Optional[float] = None
field cost: Optional[float] = None
field hallucination: Optional[float] = None
field index: int [Required]
field inputs: Dict[str, Optional[Any]] [Optional]
field like_dislike: Optional[bool] = None
field metrics: Metrics [Optional]
field prompt: Optional[str] = None
field response: Optional[str] = None
field rouge: Optional[float] = None
field target: Optional[str] = None
pydantic model PromptRows

Bases: PaginationResponseMixin

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field rows: List[PromptRow] [Optional]

promptquality.types.run module

pydantic model CreateProjectRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field name: str [Required]
Validated by:

  • generate_name

field type: str = 'prompt_evaluation'
pydantic model ProjectResponse

Bases: CreateProjectRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model BaseTemplateVersionRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field template: str [Required]
field version: Optional[int] = None
pydantic model CreateTemplateRequest

Bases: BaseTemplateVersionRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field name: str [Required]
Validated by:

  • generate_name

field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model CreateTemplateVersionRequest

Bases: BaseTemplateVersionRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field template_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model BaseTemplateVersionResponse

Bases: BaseTemplateVersionRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model CreateTemplateVersionResponse

Bases: BaseTemplateVersionResponse

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field version: int [Required]
pydantic model BaseTemplateResponse

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field all_versions: List[CreateTemplateVersionResponse] [Optional]
field id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field name: str [Required]
field selected_version: CreateTemplateVersionResponse [Required]
field selected_version_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field template: str [Required]
pydantic model BaseDatasetRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field file_path: Path [Required]
Validated by:

  • dataset_to_path

field format: DatasetFormat = DatasetFormat.csv
Validated by:

  • dataset_to_path

property files: Dict[str, BufferedReader]
pydantic model UploadDatasetRequest

Bases: BaseDatasetRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field file_path: Path [Required]
Validated by:

  • dataset_to_path

field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

Validated by:

  • dataset_to_path

field prompt_template_version_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

Validated by:

  • dataset_to_path

property params: Dict[str, str]
pydantic model UploadDatasetResponse

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field id: Annotated[UUID] [Required] (alias 'dataset_id')
Constraints:

  • uuid_version = 4

pydantic model EstimateCostRequest

Bases: BaseDatasetRequest, Settings

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

Validated by:

  • dataset_to_path

field template: str [Required]
Validated by:

  • dataset_to_path

property params: Dict[str, str]
pydantic model EstimatedCostResponse

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field log_probs_cost: float [Required]
field query_cost: float [Required]
field total_cost: float [Required]
pydantic model RunTag

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field key: Annotated[str] [Required]
Constraints:

  • max_length = 256

field tag_type: TagType [Required]
field value: Annotated[str] [Required]
Constraints:

  • max_length = 256

pydantic model CreateRunRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field name: str [Required]
Validated by:

  • generate_name

field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field run_tags: List[RunTag] [Optional]
field task_type: int = 7
pydantic model CreateRunResponse

Bases: CreateRunRequest

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model ScorerSettings

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field scorer_name: str [Required]
pydantic model ScorersConfiguration

Bases: BaseModel

Configuration to control which scorers to enable and disable.

Can be used in runs and chain runs, with or instead of scorers arg. scorers explicitly set in scorers arg will override this.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field adherence_nli: bool = False
Validated by:

  • disallow_conflicts

field chunk_attribution_utilization_gpt: bool = False
Validated by:

  • disallow_conflicts

field chunk_attribution_utilization_nli: bool = False
Validated by:

  • disallow_conflicts

field completeness_gpt: bool = False
Validated by:

  • disallow_conflicts

field completeness_nli: bool = False
Validated by:

  • disallow_conflicts

field context_relevance: bool = False
Validated by:

  • disallow_conflicts

field factuality: bool = False
Validated by:

  • disallow_conflicts

field groundedness: bool = False
Validated by:

  • disallow_conflicts

field latency: bool = True
Validated by:

  • disallow_conflicts

field pii: bool = True
Validated by:

  • disallow_conflicts

field prompt_injection: bool = False
Validated by:

  • disallow_conflicts

field prompt_perplexity: bool = False
Validated by:

  • disallow_conflicts

field sexist: bool = False
Validated by:

  • disallow_conflicts

field tone: bool = False
Validated by:

  • disallow_conflicts

field toxicity: bool = True
Validated by:

  • disallow_conflicts

classmethod from_scorers(scorers)
Return type:

ScorersConfiguration

merge_scorers(scorers)
Return type:

ScorersConfiguration

pydantic model CreateJobRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field job_name: str = 'prompt_run'
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field prompt_dataset_id: Optional[Annotated[UUID]] = None
field prompt_optimization_configuration: Optional[PromptOptimizationConfiguration] = None
field prompt_registered_scorers_configuration: Optional[List[RegisteredScorer]] = None
field prompt_scorer_settings: Optional[ScorerSettings] = None
field prompt_scorers_configuration: Optional[ScorersConfiguration] = None
field prompt_settings: Optional[Settings] = None
field prompt_template_version_id: Optional[Annotated[UUID]] = None
field run_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field task_type: Optional[int] = 7
pydantic model JobInfoMixin

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field job_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model CreateJobResponse

Bases: CreateJobRequest, JobInfoMixin

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:

pydantic model GetMetricsRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field run_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

pydantic model PromptMetrics

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field average_bleu: Optional[float] = None
field average_cost: Optional[float] = None
field average_hallucination: Optional[float] = None
field average_rouge: Optional[float] = None
field total_cost: Optional[float] = None
field total_responses: Optional[int] = None
pydantic model GetJobStatusResponse

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field error_message: Optional[str] = None
field id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field job_name: str = 'prompt_run'
field progress_message: Optional[str] = None
field progress_percent: float = 0.0
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field request_data: Optional[CreateJobRequest] = None
field run_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field status: JobStatus [Required]
field steps_completed: int = 0
field steps_total: int = 0
pydantic model CreateIntegrationRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field api_key: SecretStr [Required]
field endpoint: Optional[str] = None
field headers: Optional[Dict[str, str]] = None
field name: str [Required]
Validated by:

  • validate_name

field organization_id: Optional[str] = None
field proxy: Optional[bool] = None
serialize_api_key(value)
Return type:

str

pydantic model SelectTemplateVersionRequest

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field template_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field version: int [Required]
pydantic model UserSubmittedMetricsResponse

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field job_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field project_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field run_id: Annotated[UUID] [Required]
Constraints:

  • uuid_version = 4

field scorer_name: str [Required]

promptquality.types.settings module

pydantic model Settings

Bases: BaseModel

Settings for a prompt run that a user can configure.

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field frequency_penalty: Optional[float] = None
field max_tokens: Optional[int] = None
field model_alias: Optional[str] = None
field n: Optional[int] = None
field presence_penalty: Optional[float] = None
field response_format: Optional[Dict[str, str]] = None
field stop_sequences: Optional[List[str]] = None
field temperature: Optional[float] = None
field tools: Optional[List[Dict[str, Any]]] = None
field top_p: Optional[int] = None

promptquality.types.user_submitted_metrics module

pydantic model UserSubmittedMetrics

Bases: BaseModel

Create a new model by parsing and validating input data from keyword arguments.

Raises [ValidationError][pydantic_core.ValidationError] if the input data cannot be validated to form a valid model.

self is explicitly positional-only to allow self as a field name.

Fields:
field aggregates: Dict[str, Union[float, int, bool, str, None]] [Optional]
Validated by:

  • validate_scores_and_indices

field indices: List[int] [Optional]
Validated by:

  • validate_scores_and_indices

field name: str [Required]
Validated by:

  • validate_scores_and_indices

field scorer_name: str = '_user_submitted'
Validated by:

  • validate_scores_and_indices

field scores: List[Union[float, int, bool, str, None]] [Optional]
Validated by:

  • validate_scores_and_indices

classmethod from_scorer(scorer, prompt_rows)
Return type:

UserSubmittedMetrics

Module contents