- TypeScript 99.9%
| Filename | Latest commit message | Latest commit date |
|---|---|---|
|
|
||
| examples | ||
| scripts | ||
| src | ||
| .prettierignore | ||
| CHANGELOG.md | ||
| DEVELOPMENT.md | ||
| LICENSE | ||
| package.json | ||
| README.md | ||
| tsconfig.esm.json | ||
| tsconfig.json | ||
| vitest.config.ts | ||
Overview
@arizeai/ax-client is a TypeScript client for the Arize AX REST API. It is still under active development and is subject to change.
Arize is an AI engineering platform. It helps engineers develop, evaluate, and observe AI applications and agents.
Arize has both Enterprise and OSS products to support this goal:
- Arize AX — an enterprise AI engineering platform from development to production, with an embedded AI Copilot
- Phoenix — a lightweight, open-source project for tracing, prompt engineering, and evaluation
- OpenInference — an open-source instrumentation package to trace LLM applications across models and frameworks
Key Features
- Tracing - Trace your LLM application's runtime using OpenTelemetry-based instrumentation.
- Evaluation - Leverage LLMs to benchmark your application's performance using response and retrieval evals.
- Datasets - Create versioned datasets of examples for experimentation, evaluation, and fine-tuning.
- Experiments - Track and evaluate changes to prompts, LLMs, and retrieval.
- Playground- Optimize prompts, compare models, adjust parameters, and replay traced LLM calls.
- Prompt Management- Manage and test prompt changes systematically using version control, tagging, and experimentation.
Table of Contents
- Overview
- Key Features
- Installation
- Configuration
- Datasets
- Experiments
- Prompts
- Evaluators
- Tasks
- Spans
- Traces
- Annotation Configs
- Annotation Queues
- AI Integrations
- Integrations
- Projects
- API Keys
- Roles
- Role bindings
- Resource restrictions
- Organizations
- Organization memberships
- Users
- Spaces
- Space memberships
- Webhooks
- Webhook subscriptions
- REST endpoints
Installation
# or yarn, pnpm, bun, etc.
npm install @arizeai/ax-client
Configuration
The client will automatically read environment variables from your environment, if available.
The following environment variables are used:
ARIZE_API_KEY- The API key to use for authentication.ARIZE_BASE_URL- The base URL of the Arize AX API.
Alternatively, you can pass configuration options to the client directly, and they will be prioritized over environment variables and default values.
Datasets
The @arizeai/ax-client package allows you to create and manage datasets and their examples.
Listing datasets
import { listDatasets } from "@arizeai/ax-client";
const datasets = await listDatasets({ space: "my-space" });
console.log(datasets);
Creating a dataset
Create a dataset by providing a space (name or ID), name, and array of examples (each containing at least one property).
import { createDataset } from "@arizeai/ax-client";
const dataset = await createDataset({
name: "my-dataset",
space: "my-space",
examples: [{ question: "What is 2+2?", answer: "4", topic: "math" }],
});
Getting a dataset
import { getDataset } from "@arizeai/ax-client";
const dataset = await getDataset({ dataset: "my-dataset", space: "my-space" });
console.log(dataset);
Updating a dataset
import { updateDataset } from "@arizeai/ax-client";
const dataset = await updateDataset({
dataset: "my-dataset",
space: "my-space",
name: "my-renamed-dataset",
});
Deleting a dataset
import { deleteDataset } from "@arizeai/ax-client";
await deleteDataset({ dataset: "my-dataset", space: "my-space" });
Listing dataset examples
Lists the examples of a dataset. filter is an optional SQL-like expression
for more complex queries.
import { listDatasetExamples } from "@arizeai/ax-client";
const examples = await listDatasetExamples({
dataset: "my-dataset",
space: "my-space",
});
console.log(examples);
const filtered = await listDatasetExamples({
dataset: "my-dataset",
space: "my-space",
filter: "question = 'What is 2+2?'",
});
console.log(filtered);
Appending dataset examples
import { appendExamples } from "@arizeai/ax-client";
const result = await appendExamples({
dataset: "my-dataset",
space: "my-space",
examples: [{ question: "What is 2+2?", answer: "4", topic: "math" }],
});
console.log(result.exampleIds); // IDs of the inserted examples
Updating dataset examples
Updating examples creates a new version of the dataset.
import { updateExamples } from "@arizeai/ax-client";
const dataset = await updateExamples({
dataset: "my-dataset",
space: "my-space",
examples: [{ id: "your_example_id", question: "What is 2+2?", answer: "4" }],
newVersionName: "your_new_version_name",
});
Annotating dataset examples
import { annotateDatasetExamples } from "@arizeai/ax-client";
await annotateDatasetExamples({
space: "my-space",
dataset: "my-dataset",
annotations: [
{
recordId: "example_id_abc123",
values: [
{ name: "quality", score: 0.9 },
{ name: "topic", label: "science" },
],
},
],
});
Deleting dataset examples
Examples are removed in place from the given version; no new version is created. The delete is partial-tolerant and idempotent — the result reports which IDs were deleted and which were not.
import { deleteDatasetExamples } from "@arizeai/ax-client";
const result = await deleteDatasetExamples({
dataset: "my-dataset",
space: "my-space",
datasetVersionId: "your_dataset_version_id",
examples: ["example_id_1", "example_id_2"],
});
console.log(
result.completed,
result.deletedExampleIds,
result.notDeletedExampleIds,
);
Experiments
The @arizeai/ax-client package allows you to create and manage experiments and their runs.
Listing experiments
import { listExperiments } from "@arizeai/ax-client";
const experiments = await listExperiments({
dataset: "my-dataset",
space: "my-space",
});
console.log(experiments);
Creating an experiment
import { createExperiment } from "@arizeai/ax-client";
const experiment = await createExperiment({
experimentName: "my-experiment",
dataset: "my-dataset",
space: "my-space",
experimentRuns: [],
});
console.log(experiment);
Getting an experiment
import { getExperiment } from "@arizeai/ax-client";
// Using names
const experiment = await getExperiment({
experiment: "my-experiment",
dataset: "my-dataset",
space: "my-space",
});
// Using an ID directly
const byId = await getExperiment({ experiment: "your_experiment_id" });
Deleting an experiment
import { deleteExperiment } from "@arizeai/ax-client";
await deleteExperiment({
experiment: "my-experiment",
dataset: "my-dataset",
space: "my-space",
});
Listing experiment runs
List experiment runs by providing an experiment name or ID (space and dataset context are required when using names), optionally narrowed by an SQL-like filter over run columns, evaluations (eval.<name>.score), and annotations (annotation.<name>.*). Runs are returned in stable ID ascending order — pass pagination.nextCursor back as cursor to fetch the next page, keeping the filter unchanged while paging.
import { listExperimentRuns } from "@arizeai/ax-client";
const experimentRuns = await listExperimentRuns({
experiment: "my-experiment",
dataset: "my-dataset",
space: "my-space",
});
const filtered = await listExperimentRuns({
experiment: "my-experiment",
dataset: "my-dataset",
space: "my-space",
filter: "eval.quality.score < 0.5",
limit: 50,
});
console.log(filtered.data);
Appending experiment runs
Append between 1 and 1000 new runs to an existing experiment. Each run must include exampleId (the ID of an example from the experiment's dataset) and output. The response includes the updated experiment and the generated run IDs in input order.
import { appendExperimentRuns } from "@arizeai/ax-client";
const result = await appendExperimentRuns({
space: "my-space",
dataset: "my-dataset",
experiment: "my-experiment",
experimentRuns: [{ exampleId: "ex_abc123", output: "The answer is 42" }],
});
console.log(result.runIds); // IDs of the appended runs
Annotating experiment runs
import { annotateExperimentRuns } from "@arizeai/ax-client";
await annotateExperimentRuns({
space: "my-space",
dataset: "my-dataset",
experiment: "my-experiment",
annotations: [
{
recordId: "run_id_abc123",
values: [
{ name: "accuracy", label: "correct", score: 1.0 },
{ name: "notes", text: "Well-structured output" },
],
},
],
});
Prompts
The @arizeai/ax-client package allows you to create and manage prompts, their versions, and labels.
Creating a prompt
A prompt is created together with its initial version.
import { createPrompt } from "@arizeai/ax-client";
const prompt = await createPrompt({
space: "my-space",
name: "customer-support",
description: "A prompt for customer support interactions",
version: {
commitMessage: "Initial version",
inputVariableFormat: "F_STRING",
provider: "OPEN_AI",
model: "gpt-4",
messages: [
{ role: "SYSTEM", content: "You are a helpful assistant." },
{ role: "USER", content: "Hello, {name}!" },
],
},
});
Getting a prompt
By default the latest version is returned. Pass versionId or label to resolve a specific version instead.
import { getPrompt } from "@arizeai/ax-client";
// Latest version
const prompt = await getPrompt({
prompt: "customer-support",
space: "my-space",
});
// Version tagged with the "production" label
const productionPrompt = await getPrompt({
prompt: "customer-support",
space: "my-space",
label: "production",
});
Listing prompts
import { listPrompts } from "@arizeai/ax-client";
const { data, pagination } = await listPrompts({ space: "my-space" });
console.log(data.map((p) => p.name));
Updating a prompt
import { updatePrompt } from "@arizeai/ax-client";
const updated = await updatePrompt({
prompt: "customer-support",
space: "my-space",
description: "Updated description for the prompt",
});
Deleting a prompt
import { deletePrompt } from "@arizeai/ax-client";
await deletePrompt({ prompt: "customer-support", space: "my-space" });
Prompt versions
You can list all versions of a prompt, retrieve a specific version by its ID, and create new versions.
import {
listPromptVersions,
getPromptVersion,
createPromptVersion,
} from "@arizeai/ax-client";
// List versions
const { data: versions } = await listPromptVersions({
prompt: "customer-support",
space: "my-space",
});
// Get a specific version by its ID (pure ID, no name resolution)
const version = await getPromptVersion({ versionId: "your-version-id" });
// Create a new version
const newVersion = await createPromptVersion({
prompt: "customer-support",
space: "my-space",
commitMessage: "Updated system prompt",
inputVariableFormat: "F_STRING",
provider: "OPEN_AI",
model: "gpt-4",
messages: [
{ role: "SYSTEM", content: "You are a helpful assistant." },
{ role: "USER", content: "Hello, {name}!" },
],
});
Prompt labels
Labels let you tag a specific prompt version with a named alias (e.g. "production", "staging").
import {
getPromptVersionByLabel,
setPromptVersionLabels,
deletePromptVersionLabel,
} from "@arizeai/ax-client";
// Resolve the version a label currently points to
const version = await getPromptVersionByLabel({
prompt: "customer-support",
space: "my-space",
labelName: "production",
});
// Set labels on a version (replaces all existing labels)
const { labels } = await setPromptVersionLabels({
versionId: "your-version-id",
labels: ["production", "staging"],
});
// Remove a single label from a version
await deletePromptVersionLabel({
versionId: "your-version-id",
labelName: "staging",
});
Evaluators
The @arizeai/ax-client package allows you to create and manage LLM-as-a-judge (template), code, and remote evaluators, along with their versions.
Listing evaluators
import { listEvaluators } from "@arizeai/ax-client";
const evaluators = await listEvaluators({ space: "my-space" });
console.log(evaluators);
Creating a template evaluator
import { createTemplateEvaluator } from "@arizeai/ax-client";
const evaluator = await createTemplateEvaluator({
name: "Relevance",
space: "my-space",
commitMessage: "Initial version",
templateConfig: {
name: "Relevance",
template:
"Is the response relevant?\nQuery: {{query}}\nResponse: {{response}}",
includeExplanations: true,
useFunctionCalling: true,
classificationChoices: { relevant: 1, irrelevant: 0 },
direction: "MAXIMIZE",
llmConfig: {
aiIntegrationId: "QUlJbnRlZ3JhdGlvbjphYmMxMjM=",
modelName: "gpt-4o",
invocationParameters: { temperature: 0 },
providerParameters: {},
},
},
});
Creating a code evaluator
import { createCodeEvaluator } from "@arizeai/ax-client";
const evaluator = await createCodeEvaluator({
name: "JSON Parseable",
space: "my-space",
commitMessage: "Initial version",
codeConfig: {
type: "MANAGED",
name: "json_parseable",
managedEvaluator: "JSON_PARSEABLE",
variables: ["output"],
},
});
Creating a remote evaluator
A remote evaluator calls a customer-hosted HTTP endpoint via an EVALUATOR integration.
Requires the enableRemoteEvalTasks feature flag on the account.
import { createRemoteEvaluator } from "@arizeai/ax-client";
const evaluator = await createRemoteEvaluator({
name: "My Remote Eval",
space: "my-space",
commitMessage: "Initial version",
integrationId: "<evaluator-integration-id>",
});
Getting an evaluator
import { getEvaluator } from "@arizeai/ax-client";
const evaluator = await getEvaluator({
evaluator: "Relevance",
space: "my-space",
});
console.log(evaluator);
Updating an evaluator
import { updateEvaluator } from "@arizeai/ax-client";
const evaluator = await updateEvaluator({
evaluator: "Relevance",
space: "my-space",
name: "Updated Evaluator Name",
});
Deleting an evaluator
import { deleteEvaluator } from "@arizeai/ax-client";
await deleteEvaluator({ evaluator: "Relevance", space: "my-space" });
Evaluator versions
You can list all versions of an evaluator, retrieve a specific version by its ID, and create new versions.
import {
listEvaluatorVersions,
getEvaluatorVersion,
createTemplateEvaluatorVersion,
} from "@arizeai/ax-client";
// List versions
const versions = await listEvaluatorVersions({
evaluator: "Relevance",
space: "my-space",
});
// Get a specific version by its ID
const version = await getEvaluatorVersion({ versionId: "your-version-id" });
// Create a new template version (use createCodeEvaluatorVersion for code evaluators,
// or createRemoteEvaluatorVersion for remote evaluators)
const newVersion = await createTemplateEvaluatorVersion({
evaluator: "Relevance",
space: "my-space",
commitMessage: "Updated prompt template",
templateConfig: {
name: "Relevance",
template: "Rate the relevance.\nQuery: {{query}}\nResponse: {{response}}",
includeExplanations: true,
useFunctionCalling: true,
classificationChoices: { relevant: 1, irrelevant: 0 },
direction: "MAXIMIZE",
llmConfig: {
aiIntegrationId: "QUlJbnRlZ3JhdGlvbjphYmMxMjM=",
modelName: "gpt-4o",
invocationParameters: { temperature: 0 },
providerParameters: {},
},
},
});
// Create a new remote version (switches to a different EVALUATOR integration)
import { createRemoteEvaluatorVersion } from "@arizeai/ax-client";
const remoteVersion = await createRemoteEvaluatorVersion({
evaluator: "My Remote Eval",
space: "my-space",
commitMessage: "Switch endpoint",
integrationId: "<evaluator-integration-id>",
});
Deleting evaluator versions
import { deleteEvaluatorVersions } from "@arizeai/ax-client";
await deleteEvaluatorVersions({
evaluator: "Relevance",
space: "my-space",
versionIds: ["your_version_id_1", "your_version_id_2"],
});
Tasks
The @arizeai/ax-client package allows you to create and manage scheduled or on-demand tasks (server-side evaluations and experiments) and monitor their runs.
Tip: Prefer the narrowly-typed
createEvaluationTaskandcreateRunExperimentTaskhelpers shown below. A lower-levelcreateTaskis also exported if you need to construct a request payload directly.
Listing tasks
import { listTasks } from "@arizeai/ax-client";
const tasks = await listTasks({ space: "my-space", name: "prod" });
console.log(tasks);
Creating an evaluation task
Each evaluator is either a span-shape entry (with columnMappings / queryFilter) or a
trace/session-shape entry (with queryMappings). Supply one shape per evaluator; they are
mutually exclusive via the SpanEvaluatorInput | TraceOrSessionEvaluatorInput union.
Span-granularity task:
import { createEvaluationTask } from "@arizeai/ax-client";
const task = await createEvaluationTask({
name: "Weekly Quality Check",
type: "TEMPLATE_EVALUATION",
space: "my-space",
project: "my-project",
queryFilter: "span_kind = 'LLM'",
evaluators: [
{
evaluatorId: "your_evaluator_id",
columnMappings: {
input: "attributes.input.value",
output: "attributes.output.value",
},
},
],
});
Trace/session-granularity task (multi-span query):
const task = await createEvaluationTask({
name: "Trace Quality Check",
type: "TEMPLATE_EVALUATION",
space: "my-space",
project: "my-project",
queryFilters: {
filters: [
{ id: "A", filter: "span_kind = 'LLM'" },
{ id: "B", filter: "span_kind = 'RETRIEVER'" },
],
expression: "A AND B",
},
evaluators: [
{
evaluatorId: "your_evaluator_id",
queryMappings: [
{
variableName: "input",
queryIds: ["A"],
attributePath: "attributes.input.value",
},
{
variableName: "output",
queryIds: ["B"],
attributePath: "attributes.output.value",
},
],
},
],
});
Creating a run-experiment task
import { createRunExperimentTask } from "@arizeai/ax-client";
const task = await createRunExperimentTask({
name: "GPT-4o Baseline Task",
dataset: "my-dataset",
space: "my-space",
runConfiguration: {
experiment_type: "LLM_GENERATION",
aiIntegration: "my-openai-integration",
model_name: "gpt-4o",
input_variable_format: "F_STRING",
messages: [
{ role: "SYSTEM", content: "You are a helpful assistant." },
{ role: "USER", content: "Answer: {question}" },
],
},
});
Getting a task
import { getTask } from "@arizeai/ax-client";
// By ID
const task = await getTask({ task: "your_task_id" });
// By name (requires space)
const byName = await getTask({ task: "My Task", space: "my-space" });
Updating and deleting a task
import { updateTask, deleteTask } from "@arizeai/ax-client";
// Basic update — rename and adjust sampling rate
await updateTask({
task: "your_task_id",
name: "Renamed Task",
samplingRate: 0.5,
});
// Switch to trace/session shape (multi-span query)
await updateTask({
task: "your_task_id",
queryFilters: {
filters: [{ id: "A", filter: "span_kind = 'LLM'" }],
expression: "A",
},
evaluators: [
{
evaluatorId: "your_evaluator_id",
queryMappings: [
{
variableName: "input",
queryIds: ["A"],
attributePath: "attributes.input.value",
},
],
},
],
});
await deleteTask({ task: "your_task_id" });
Triggering and monitoring task runs
import {
triggerTaskRun,
waitForTaskRun,
listTaskRuns,
getTaskRun,
cancelTaskRun,
} from "@arizeai/ax-client";
// Trigger a run, then poll until it finishes
const run = await triggerTaskRun({ task: "My Task", space: "my-space" });
const finalRun = await waitForTaskRun({
runId: run.id,
pollInterval: 3_000, // poll every 3 seconds
timeout: 5 * 60_000, // give up after 5 minutes
});
console.log(finalRun.status); // "COMPLETED" | "FAILED" | "CANCELLED"
// List, inspect, or cancel runs
const { data: runs } = await listTaskRuns({
task: "My Task",
space: "my-space",
});
const single = await getTaskRun({ runId: run.id });
await cancelTaskRun({ runId: run.id });
Spans
The @arizeai/ax-client package allows you to list, annotate, and delete spans within a project.
Listing spans
import { listSpans } from "@arizeai/ax-client";
// By project ID
const spans = await listSpans({ project: "your_project_id" });
// By project name (requires space)
const byName = await listSpans({ project: "My Project", space: "my-space" });
Column projection
Set includedColumns to return only selected columns. Set excludedColumns
to omit selected columns, such as a large embedding vector. Do not set both.
Fixed span fields are always returned. Set truncateStringLength to shorten
long string values inside each span's attributes object. Strings longer
than the limit are cut and end with …, which counts toward the limit. Span
IDs, names, status, timestamps, annotations, evaluations, and events are not
truncated.
const spans = await listSpans({
project: "your_project_id",
excludedColumns: ["attributes.embedding.vectors"],
truncateStringLength: 200,
});
Annotating spans
granularity selects what each recordId identifies: a span (SPAN, the
default), a trace's root span (TRACE), or a session (SESSION, written to
the root span of the session's earliest trace). Up to 1000 records may be
annotated per request for SPAN/TRACE; up to 100 for SESSION.
import { annotateSpans } from "@arizeai/ax-client";
await annotateSpans({
space: "my-space",
project: "my-project",
annotations: [
{
recordId: "c3Bhbl9pZF9hYmMxMjM=", // base64-encoded span ID
values: [
{ name: "quality", score: 0.9 },
{ name: "topic", label: "science" },
],
},
],
});
startTime/endTime bound the lookup window and are both optional: when
omitted, endTime defaults to now and startTime defaults to 31 days before
that (7 days for SESSION). If the record is older than the default window,
pass startTime explicitly or the record won't be found.
// Session-granularity example
await annotateSpans({
space: "my-space",
project: "my-project",
granularity: "SESSION",
startTime: new Date(Date.now() - 7 * 24 * 60 * 60 * 1000),
endTime: new Date(),
annotations: [
{
recordId: "20144", // session ID
values: [{ name: "quality", label: "good" }],
},
],
});
Deleting spans
import { deleteSpans } from "@arizeai/ax-client";
// By project name (requires space)
await deleteSpans({
project: "My Project",
space: "my-space",
spanIds: ["a1b2c3d4e5f6a7b8", "f8e7d6c5b4a39281"],
});
Traces
The @arizeai/ax-client package allows you to list traces within a project. Each trace carries its full (flat) list of spans plus lightweight roll-up metadata; reconstruct the trace tree client-side using each span's parentId.
Listing traces
import { listTraces } from "@arizeai/ax-client";
// By project ID
const { data: traces, pagination } = await listTraces({
project: "your_project_id",
});
// By project name (requires space), with a filter and time range
const filtered = await listTraces({
project: "My Project",
space: "my-space",
filter: "status_code = 'ERROR'",
startTime: new Date("2024-01-01T00:00:00Z"),
endTime: new Date("2024-01-02T00:00:00Z"),
limit: 50,
});
// Page through results using the opaque cursor
const next = await listTraces({
project: "your_project_id",
cursor: pagination.nextCursor,
});
The filter uses the same SQL-like syntax as listSpans, but the semantics are
trace-contains-match: a trace is returned when any of its spans matches the
filter (the matching span is usually a child, not the root).
Each response page also has an overall span limit, so a trace's span list can
be incomplete even when spansTruncated is false. Narrow the time window or
call listSpans with a trace_id filter to retrieve the trace's spans directly.
Annotation Configs
The @arizeai/ax-client package allows you to create and manage annotation configs, which define the annotations that can be applied to spans and examples.
Listing annotation configs
import { listAnnotationConfigs } from "@arizeai/ax-client";
const annotationConfigs = await listAnnotationConfigs({ space: "my-space" });
console.log(annotationConfigs);
Creating an annotation config
import { createAnnotationConfig } from "@arizeai/ax-client";
const annotationConfig = await createAnnotationConfig({
name: "Accuracy",
space: "my-space",
type: "CATEGORICAL",
values: [
{ label: "accurate", score: 1 },
{ label: "inaccurate", score: 0 },
],
optimizationDirection: "MAXIMIZE",
});
Getting an annotation config
import { getAnnotationConfig } from "@arizeai/ax-client";
const annotationConfig = await getAnnotationConfig({
annotationConfig: "Accuracy",
space: "my-space",
});
Updating an annotation config
There is a dedicated update function per annotation config type. Each one must be called with the stored config's type — a config's type is immutable and cannot be changed. Any fields you omit are left unchanged.
import { updateCategoricalAnnotationConfig } from "@arizeai/ax-client";
const annotationConfig = await updateCategoricalAnnotationConfig({
annotationConfig: "Accuracy",
space: "my-space",
name: "Accuracy v2",
values: [
{ label: "accurate", score: 1 },
{ label: "inaccurate", score: 0 },
],
optimizationDirection: "MAXIMIZE",
});
import { updateContinuousAnnotationConfig } from "@arizeai/ax-client";
const annotationConfig = await updateContinuousAnnotationConfig({
annotationConfig: "Accuracy",
space: "my-space",
name: "Accuracy v2",
minimumScore: 0,
maximumScore: 10,
optimizationDirection: "MAXIMIZE",
});
import { updateFreeformAnnotationConfig } from "@arizeai/ax-client";
const annotationConfig = await updateFreeformAnnotationConfig({
annotationConfig: "Notes",
space: "my-space",
name: "Notes v2",
});
Deleting an annotation config
import { deleteAnnotationConfig } from "@arizeai/ax-client";
await deleteAnnotationConfig({
annotationConfig: "Accuracy",
space: "my-space",
});
Annotation Queues
The @arizeai/ax-client package allows you to create and manage annotation queues, add records to them, and annotate or assign those records.
Listing annotation queues
import { listAnnotationQueues } from "@arizeai/ax-client";
const annotationQueues = await listAnnotationQueues({ space: "my-space" });
console.log(annotationQueues);
Creating an annotation queue
import { createAnnotationQueue } from "@arizeai/ax-client";
const queue = await createAnnotationQueue({
name: "Quality Review Queue",
spaceId: "your_space_id",
annotationConfigIds: ["ac_abc123"],
assignmentMethod: "ALL",
});
Getting, updating, and deleting a queue
import {
getAnnotationQueue,
updateAnnotationQueue,
deleteAnnotationQueue,
} from "@arizeai/ax-client";
const queue = await getAnnotationQueue({
annotationQueue: "my_queue",
space: "my-space",
});
await updateAnnotationQueue({
annotationQueue: "my_queue",
space: "my-space",
name: "Updated Queue Name",
});
await deleteAnnotationQueue({ annotationQueue: "my_queue", space: "my-space" });
Managing queue records
import {
addAnnotationQueueRecords,
listAnnotationQueueRecords,
deleteAnnotationQueueRecords,
annotateAnnotationQueueRecord,
assignAnnotationQueueRecord,
} from "@arizeai/ax-client";
// Add records to the queue from span and session sources
await addAnnotationQueueRecords({
annotationQueue: "my_queue",
space: "my-space",
recordSources: [
{
recordType: "SPAN",
projectId: "proj_abc123",
startTime: "2024-01-15T00:00:00Z",
endTime: "2024-01-15T23:59:59Z",
spanIds: ["span_abc123"],
},
{
recordType: "SESSION",
projectId: "proj_abc123",
startTime: "2024-01-15T00:00:00Z",
endTime: "2024-01-15T23:59:59Z",
sessionIds: ["session_abc123"],
},
],
});
// List records
const records = await listAnnotationQueueRecords({
annotationQueue: "my_queue",
space: "my-space",
});
// Assign a record to the current user
await assignAnnotationQueueRecord({
annotationQueue: "my_queue",
space: "my-space",
annotationQueueRecordId: "aqr_abc123",
});
// Submit annotations for a record
await annotateAnnotationQueueRecord({
annotationQueue: "my_queue",
space: "my-space",
annotationQueueRecordId: "aqr_abc123",
annotations: [
{ name: "accuracy", label: "correct", score: 1.0 },
{ name: "quality", text: "Well-structured response" },
],
});
// Remove records
await deleteAnnotationQueueRecords({
annotationQueue: "my_queue",
space: "my-space",
recordIds: ["aqr_abc123", "aqr_def456"],
});
AI Integrations
The @arizeai/ax-client package allows you to create and manage AI integrations (LLM provider credentials used by evaluators, tasks, and the playground).
Listing AI integrations
import { listAiIntegrations } from "@arizeai/ax-client";
const integrations = await listAiIntegrations({ space: "my-space" });
console.log(integrations);
Creating an AI integration
import { createAiIntegration } from "@arizeai/ax-client";
const integration = await createAiIntegration({
name: "Production OpenAI",
provider: "OPEN_AI",
apiKey: "sk-...",
modelNames: ["gpt-4o", "gpt-4o-mini"],
enableDefaultModels: true,
});
Getting an AI integration
import { getAiIntegration } from "@arizeai/ax-client";
// By ID
const integration = await getAiIntegration({
integration: "your_integration_id",
});
// By name (requires space)
const byName = await getAiIntegration({
integration: "Production OpenAI",
space: "my-space",
});
Updating an AI integration
import { updateAiIntegration } from "@arizeai/ax-client";
const integration = await updateAiIntegration({
integration: "Production OpenAI",
space: "my-space",
name: "Updated OpenAI",
modelNames: ["gpt-4o"],
});
Deleting an AI integration
import { deleteAiIntegration } from "@arizeai/ax-client";
await deleteAiIntegration({
integration: "Production OpenAI",
space: "my-space",
});
Integrations
The @arizeai/ax-client package allows you to create and manage integrations on the polymorphic /v2/integrations surface, covering both LLM (model-provider) and AGENT (customer-hosted endpoint) integrations. Integration names are unique per account and type, so type is required when resolving a name; an ID always works on its own. These functions are in alpha and emit a one-time pre-release warning.
Listing integrations
type is an optional filter. Omit it to list integrations of every type in one call; each item carries its type discriminator.
import { listIntegrations } from "@arizeai/ax-client";
// Only agent integrations
const agents = await listIntegrations({ type: "AGENT", space: "my-space" });
// Every type, discriminated by `integration.type`
const all = await listIntegrations({});
Creating an integration
The type field selects the config shape.
import { createIntegration } from "@arizeai/ax-client";
// LLM integration
const llm = await createIntegration({
type: "LLM",
name: "Production OpenAI",
config: { provider: "OPEN_AI", apiKey: "sk-..." },
});
// Agent integration
const agent = await createIntegration({
type: "AGENT",
name: "My Support Agent",
config: {
endpoint: "https://agent.example.com/replay",
inputSchema: { type: "object", properties: { input: { type: "string" } } },
requestPresets: [{ name: "default", config: { input: "hello" } }],
},
});
config.provider selects the LLM config shape. Secrets are write-only and never returned.
| Provider | Required fields | Optional fields |
|---|---|---|
OPEN_AI, ANTHROPIC, GEMINI |
apiKey |
isFunctionCallingEnabled |
AWS_BEDROCK |
auth |
modelNames, isDefaultModelsEnabled |
VERTEX_AI |
projectId, location, projectAccessLabel |
|
CUSTOM |
baseUrl |
apiKey, headers, modelNames, isDefaultModelsEnabled, isFunctionCallingEnabled |
NVIDIA_NIM |
none | baseUrl, apiKey, headers, modelNames, isDefaultModelsEnabled, isFunctionCallingEnabled |
LITELLM |
baseUrl, apiKey |
headers, modelNames, isFunctionCallingEnabled |
FIREWORKS, TOGETHER_AI |
apiKey |
modelNames, isDefaultModelsEnabled, isFunctionCallingEnabled |
// Self-hosted LiteLLM proxy: the endpoint and virtual key are both required.
const litellm = await createIntegration({
type: "LLM",
name: "Team LiteLLM",
config: {
provider: "LITELLM",
baseUrl: "https://litellm.internal:4000",
apiKey: "sk-...",
},
});
// Hosted providers such as Fireworks AI and Together AI need only the API key.
// Arize resolves the models the key can reach; list `modelNames` for
// fine-tunes or dedicated deployments that cannot be discovered.
const together = await createIntegration({
type: "LLM",
name: "Together AI",
config: { provider: "TOGETHER_AI", apiKey: "..." },
});
Getting an integration
import { getIntegration } from "@arizeai/ax-client";
// By ID
const integration = await getIntegration({
integration: "your_integration_id",
});
// By name (requires type)
const byName = await getIntegration({
integration: "Production OpenAI",
type: "LLM",
});
Updating an integration
type is required (it selects the update shape) and is immutable server-side. Provide at least one updatable field; omitted fields are preserved.
import { updateIntegration } from "@arizeai/ax-client";
const integration = await updateIntegration({
integration: "Production OpenAI",
type: "LLM",
config: { provider: "OPEN_AI", apiKey: "sk-new-key" },
});
Deleting an integration
import { deleteIntegration } from "@arizeai/ax-client";
await deleteIntegration({
integration: "My Support Agent",
type: "AGENT",
});
Projects
The @arizeai/ax-client package allows you to create and manage projects.
Creating a project
import { createProject } from "@arizeai/ax-client";
const project = await createProject({
space: "my-space",
name: "my-project",
});
console.log(project);
Getting a project
import { getProject } from "@arizeai/ax-client";
const project = await getProject({
project: "my-project",
space: "my-space",
});
console.log(project);
Listing projects
import { listProjects } from "@arizeai/ax-client";
const { projects } = await listProjects({ space: "my-space" });
console.log(projects);
Updating a project
import { updateProject } from "@arizeai/ax-client";
const project = await updateProject({
project: "my-project",
space: "my-space",
name: "my-renamed-project",
});
console.log(project);
Deleting a project
import { deleteProject } from "@arizeai/ax-client";
await deleteProject({ project: "my-project", space: "my-space" });
API Keys
The @arizeai/ax-client package allows you to create, list, delete, revoke, and refresh API keys.
Creating an API key
import { createApiKey } from "@arizeai/ax-client";
const apiKey = await createApiKey({ keyType: "USER", name: "CI pipeline key" });
// Store apiKey.key securely — the full key value is only returned once
console.log(apiKey.key);
Service keys are backed by a dedicated bot user scoped to one or more organizations and spaces:
const apiKey = await createApiKey({
keyType: "SERVICE",
name: "service-key",
organizations: [
{
orgId: "T3JnMTIz",
spaces: [{ spaceId: "U3BhY2UxMjM", role: { name: "MEMBER" } }],
},
],
expiresAt: new Date("2027-01-01"),
});
// apiKey.botUser — the bot user created for this key, with resolved role assignments
Listing API keys
import { listApiKeys } from "@arizeai/ax-client";
const { data } = await listApiKeys();
console.log(data.map((k) => k.name));
// Filter by key type or status
const { data: userKeys } = await listApiKeys({ keyType: "USER" });
// List service keys for a specific space
const { data: serviceKeys } = await listApiKeys({
keyType: "SERVICE",
spaceId: "U3BhY2U6MTIzNDU=",
});
// List service keys created by a specific user (any caller with space access)
const { data: byUser } = await listApiKeys({
spaceId: "U3BhY2U6MTIzNDU=",
userId: "VXNlcjoxMjM0NQ==",
});
// Each service key entry carries the same botUser detail createApiKey returns —
// use botUser.id as the userId when creating a role binding for the key.
for (const key of serviceKeys) {
console.log(key.botUser?.id);
}
Revoking an API key
Sets the key's status to revoked and deactivates it immediately. This operation is irreversible; revoking an already-revoked key is a no-op and still succeeds.
import { revokeApiKey } from "@arizeai/ax-client";
await revokeApiKey({ apiKeyId: "your-api-key-id" });
Refreshing an API key
Atomically revokes an existing key and issues a replacement with the same metadata (name, description, key type). The old key is invalidated and the new key is activated in a single transaction.
import { refreshApiKey } from "@arizeai/ax-client";
const refreshed = await refreshApiKey({ apiKeyId: "your-api-key-id" });
// Store refreshed.key securely — the full key value is only returned once
console.log(refreshed.key);
Supply expiresAt to set an expiration on the replacement key:
const refreshed = await refreshApiKey({
apiKeyId: "your-api-key-id",
expiresAt: new Date("2027-01-01"),
});
Supply gracePeriodSeconds to keep the old key valid for a window after the refresh, giving your services time to adopt the new key before the old one is invalidated:
const refreshed = await refreshApiKey({
apiKeyId: "your-api-key-id",
gracePeriodSeconds: 3600, // old key stays valid for 1 hour
});
// Store refreshed.key securely — the full key value is only returned once
console.log(refreshed.key);
Roles
The @arizeai/ax-client package allows you to create and manage custom roles.
Creating a role
import { createRole } from "@arizeai/ax-client";
const role = await createRole({
name: "AI Engineer",
permissions: ["PROJECT_READ", "DATASET_READ", "DATASET_CREATE"],
description: "Can read and create datasets and experiments.",
});
Listing roles
import { listRoles } from "@arizeai/ax-client";
const { data } = await listRoles();
console.log(data.map((r) => r.name));
// Filter to only predefined (system) roles
const { data: predefined } = await listRoles({ isPredefined: true });
Getting a role
import { getRole } from "@arizeai/ax-client";
const role = await getRole({ roleId: "your-role-id" });
Updating a role
import { updateRole } from "@arizeai/ax-client";
const role = await updateRole({
roleId: "your-role-id",
permissions: ["PROJECT_READ", "DATASET_READ"],
});
Deleting a role
import { deleteRole } from "@arizeai/ax-client";
await deleteRole({ roleId: "your-role-id" });
Role bindings
The @arizeai/ax-client package allows you to bind a role to a user on a specific resource (e.g. a project or space).
Creating a role binding
import { createRoleBinding } from "@arizeai/ax-client";
const binding = await createRoleBinding({
userId: "VXNlcjoxMjM0NQ==",
roleId: "Um9sZTphYmMxMjM=",
resourceType: "PROJECT",
resourceId: "UHJvamVjdDphYmMxMjM=",
});
console.log(binding);
userId is the ID of the user to bind the role to. To bind a service key, pass its bot
user's ID instead of your own — it is returned as botUser.id from createApiKey (see
Creating an API key above).
Listing role bindings
import { listRoleBindings } from "@arizeai/ax-client";
const bindings = await listRoleBindings({ resourceType: "SPACE" });
console.log(bindings);
Getting a role binding
import { getRoleBinding } from "@arizeai/ax-client";
const binding = await getRoleBinding({ bindingId: "Um9sZUJpbmRpbmc6YWJjMTIz" });
Updating a role binding
import { updateRoleBinding } from "@arizeai/ax-client";
const binding = await updateRoleBinding({
bindingId: "Um9sZUJpbmRpbmc6YWJjMTIz",
roleId: "Um9sZTphYmMxMjM=",
});
Deleting a role binding
import { deleteRoleBinding } from "@arizeai/ax-client";
await deleteRoleBinding({ bindingId: "Um9sZUJpbmRpbmc6YWJjMTIz" });
Resource restrictions
The @arizeai/ax-client package allows you to restrict access to a resource so that only explicitly bound users can access it, and to lift that restriction.
import { restrictResource, unrestrictResource } from "@arizeai/ax-client";
// Restrict a resource
const restriction = await restrictResource({ resourceId: "your_project_id" });
console.log(restriction);
// Lift the restriction
await unrestrictResource({ resourceId: "your_project_id" });
Organizations
The @arizeai/ax-client package allows you to manage organizations.
Listing organizations
import { listOrganizations } from "@arizeai/ax-client";
const result = await listOrganizations();
console.log(result.organizations);
Getting an organization
import { getOrganization } from "@arizeai/ax-client";
const org = await getOrganization({ organization: "your-organization-name" });
console.log(org);
Creating an organization
import { createOrganization } from "@arizeai/ax-client";
const org = await createOrganization({
name: "your-organization-name",
description: "Optional description",
});
console.log(org);
Updating an organization
import { updateOrganization } from "@arizeai/ax-client";
const org = await updateOrganization({
organization: "your-organization-name",
name: "your-organization-name-updated",
});
console.log(org);
Deleting an organization
Warning: This operation is irreversible and deletes the organization and all resources that belong to it, including all spaces and their contents.
import { deleteOrganization } from "@arizeai/ax-client";
await deleteOrganization({ organization: "your-organization-name" });
Organization memberships
Adding a user to an organization
import { addOrganizationUser } from "@arizeai/ax-client";
const membership = await addOrganizationUser({
organizationId: "org_abc123",
userId: "VXNlcjoxMjM0NQ==",
role: { type: "PREDEFINED", name: "MEMBER" },
});
console.log(membership);
Removing a user from an organization
import { removeOrganizationUser } from "@arizeai/ax-client";
await removeOrganizationUser({
organizationId: "org_abc123",
userId: "VXNlcjoxMjM0NQ==",
});
Users
The @arizeai/ax-client package allows you to manage users.
Note: Unlike organizations, users are identified by opaque ID only — not by name. User display names are not unique within an account, so all functions require the user's ID.
Listing users
import { listUsers } from "@arizeai/ax-client";
const result = await listUsers();
console.log(result.data);
Getting a user
import { getUser } from "@arizeai/ax-client";
const user = await getUser({ userId: "VXNlcjoxMjM0NQ==" });
console.log(user);
Creating a user
The inviteMode parameter controls how the user is invited:
"EMAIL_LINK"— sends the user an email with a verification link."TEMPORARY_PASSWORD"— issues a one-time password returned in the response."NONE"— pre-provisions an SSO user directly; no invitation is sent and the user is immediately active via the configured identity provider.
Returns a UserCreated object (HTTP 201) for a new user, or a User object
(HTTP 200) when an existing invited user is returned as-is (idempotent — the
invitation is not resent).
import { createUser } from "@arizeai/ax-client";
const user = await createUser({
name: "Jane Smith",
email: "jane.smith@example.com",
role: { type: "PREDEFINED", name: "MEMBER" },
inviteMode: "EMAIL_LINK",
});
console.log(user);
Updating a user
import { updateUser } from "@arizeai/ax-client";
const user = await updateUser({
userId: "VXNlcjoxMjM0NQ==",
name: "Jane Smith Updated",
isDeveloper: true,
});
console.log(user);
Deleting a user
Warning: This operation permanently blocks the user from the account. Blocked users cannot be re-invited — inactive is a terminal state. The operation cascades to organization memberships, space memberships, API keys, and role bindings.
import { deleteUser } from "@arizeai/ax-client";
await deleteUser({ userId: "VXNlcjoxMjM0NQ==" });
Resending a user invitation
import { resendInvitation } from "@arizeai/ax-client";
await resendInvitation({ userId: "VXNlcjoxMjM0NQ==" });
Resetting a user's password
import { resetPassword } from "@arizeai/ax-client";
await resetPassword({ userId: "VXNlcjoxMjM0NQ==" });
Bulk deleting users
Delete multiple users by ID and/or email. Each entry's outcome is returned individually.
import { bulkDeleteUsers } from "@arizeai/ax-client";
const results = await bulkDeleteUsers({
userIds: ["VXNlcjoxMjM0NQ=="],
emails: ["jane.smith@example.com"],
});
for (const r of results) {
console.log(r.userId, r.status, r.error ?? "");
}
Spaces
The @arizeai/ax-client package allows you to create and manage spaces.
Space responses from list, get, create, and update include organizationId,
the ID of the parent organization, when the server supports this field.
Older server versions may omit it.
Listing spaces
import { listSpaces } from "@arizeai/ax-client";
const spaces = await listSpaces({
organizationId: "QWNjb3VudE9yZ2FuaXphdGlvbjox",
});
console.log(spaces);
Omit organizationId to list accessible spaces from all organizations. The SDK
sends this value as the REST query parameter org_id.
Getting a space
import { getSpace } from "@arizeai/ax-client";
const space = await getSpace({ space: "my-space" });
console.log(space);
Creating a space
import { createSpace } from "@arizeai/ax-client";
const space = await createSpace({
organizationId: "T3JnYW5pemF0aW9uOmFiYzEyMw==",
name: "your_space_name",
isPrivate: true, // omit or set false for a public space (default)
});
Private spaces are visible only to their members and account/org/space admins.
A warning is logged when isPrivate: true to remind you to add members before
the space becomes inaccessible to other users.
Updating a space
import { updateSpace } from "@arizeai/ax-client";
const space = await updateSpace({
space: "my-space",
name: "updated_space_name",
isPrivate: false, // omit to preserve the current visibility
});
Deleting a space
import { deleteSpace } from "@arizeai/ax-client";
await deleteSpace({ space: "my-space" });
Space memberships
Adding a user to a space
import { addSpaceUser } from "@arizeai/ax-client";
const membership = await addSpaceUser({
spaceId: "spc_abc123",
userId: "VXNlcjoxMjM0NQ==",
role: { type: "PREDEFINED", name: "MEMBER" },
});
console.log(membership);
Removing a user from a space
import { removeSpaceUser } from "@arizeai/ax-client";
await removeSpaceUser({
spaceId: "spc_abc123",
userId: "VXNlcjoxMjM0NQ==",
});
Webhooks
Webhooks are alpha: every call logs a one-time warning and the API may change. A webhook is an organization-owned HTTPS destination. Pass a webhook ID, or a name together with organization.
Listing webhooks
import { listWebhooks } from "@arizeai/ax-client";
const webhooks = await listWebhooks({ organization: "my-org" });
console.log(webhooks.data);
Creating a webhook
import { createWebhook } from "@arizeai/ax-client";
const webhook = await createWebhook({
organization: "my-org",
name: "deploy-notifier",
url: "https://example.com/hooks/arize",
authType: "HMAC_SHA256",
});
// Only returned here. Store it now; afterwards only signingSecretHint is readable.
console.log(webhook.signingSecret);
authToken and headers are write-only and never come back in any response.
Getting, updating, testing, and deleting a webhook
import {
getWebhook,
updateWebhook,
testWebhook,
deleteWebhook,
} from "@arizeai/ax-client";
const webhook = await getWebhook({
webhook: "deploy-notifier",
organization: "my-org",
});
// Omitted fields keep their value; pass `description: null` to clear it.
await updateWebhook({
webhook: "deploy-notifier",
organization: "my-org",
timeoutMs: 10000,
});
const outcome = await testWebhook({
webhook: "deploy-notifier",
organization: "my-org",
});
console.log(outcome.statusCode, outcome.errorMessage);
await deleteWebhook({ webhook: "deploy-notifier", organization: "my-org" });
Listing delivery attempts
import { listWebhookDeliveryAttempts } from "@arizeai/ax-client";
const attempts = await listWebhookDeliveryAttempts({
webhook: "deploy-notifier",
organization: "my-org",
limit: 100, // up to 500
});
console.log(attempts.data);
Webhook subscriptions
A subscription delivers one event from one prompt or evaluator to one webhook. Subscriptions have no name and are addressed by ID.
Subscribing a webhook to an event
import { createWebhookSubscription } from "@arizeai/ax-client";
const subscription = await createWebhookSubscription({
webhook: "deploy-notifier",
organization: "my-org",
sourceType: "PROMPT",
sourceId: "your_prompt_id",
event: "PROMPT_VERSION_CREATED",
});
Listing, getting, and deleting subscriptions
import {
listWebhookSubscriptions,
getWebhookSubscription,
deleteWebhookSubscription,
} from "@arizeai/ax-client";
// sourceType and sourceId go together; omit both to list every readable subscription.
const subscriptions = await listWebhookSubscriptions({
sourceType: "PROMPT",
sourceId: "your_prompt_id",
});
const subscription = await getWebhookSubscription({
subscriptionId: "your_subscription_id",
});
await deleteWebhookSubscription({ subscriptionId: "your_subscription_id" });
REST endpoints
It is recommended to use the methods in this package. If more control is desired, you can use the client directly. The client provides a type-safe fetch for the entire Arize AX REST API.
import { createClient } from "@arizeai/ax-client";
const client = createClient();
// Get all datasets
const response = await client.GET("/v2/datasets");