ChatOpenAI

This will help you getting started with ChatOpenAI chat models. For detailed documentation of all ChatOpenAI features and configurations head to the API reference.

Overview

Integration details

Class	Package	Local	Serializable	PY support	Package downloads	Package latest
ChatOpenAI	@langchain/openai	❌	✅	✅

Model features

Tool calling	Structured output	JSON mode	Image input	Audio input	Video input	Token-level streaming	Token usage	Logprobs
✅	✅	✅	✅	❌	❌	✅	✅	✅

Setup

To access ChatOpenAI models you’ll need to create a ChatOpenAI account, get an API key, and install the @langchain/openai integration package.

Credentials

Head to OpenAI’s website to sign up to ChatOpenAI and generate an API key. Once you’ve done this set the OPENAI_API_KEY environment variable:

export OPENAI_API_KEY="your-api-key"

If you want to get automated tracing of your model calls you can also set your LangSmith API key by uncommenting below:

# export LANGCHAIN_TRACING_V2="true"
# export LANGCHAIN_API_KEY="your-api-key"

Installation

The LangChain ChatOpenAI integration lives in the @langchain/openai package:

tip

See this section for general instructions on installing integration packages.

npm
yarn
pnpm

npm i @langchain/openai

yarn add @langchain/openai

pnpm add @langchain/openai

Instantiation

Now we can instantiate our model object and generate chat completions:

import { ChatOpenAI } from "@langchain/openai";

const llm = new ChatOpenAI({
  model: "gpt-4o",
  temperature: 0,
  maxTokens: undefined,
  timeout: undefined,
  maxRetries: 2,
  // other params...
});

Invocation

const aiMsg = await llm.invoke([
  [
    "system",
    "You are a helpful assistant that translates English to French. Translate the user sentence.",
  ],
  ["human", "I love programming."],
]);
aiMsg;

AIMessage {
  "id": "chatcmpl-9rB4GvhlRb0x3hxupLBQYOKKmTxvV",
  "content": "J'adore la programmation.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 8,
      "promptTokens": 31,
      "totalTokens": 39
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 31,
    "output_tokens": 8,
    "total_tokens": 39
  }
}

console.log(aiMsg.content);

J'adore la programmation.

Chaining

We can chain our model with a prompt template like so:

import { ChatPromptTemplate } from "@langchain/core/prompts";

const prompt = ChatPromptTemplate.fromMessages([
  [
    "system",
    "You are a helpful assistant that translates {input_language} to {output_language}.",
  ],
  ["human", "{input}"],
]);

const chain = prompt.pipe(llm);
await chain.invoke({
  input_language: "English",
  output_language: "German",
  input: "I love programming.",
});

AIMessage {
  "id": "chatcmpl-9rB4JD9rVBLzTuMee9AabulowEH0d",
  "content": "Ich liebe das Programmieren.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 6,
      "promptTokens": 26,
      "totalTokens": 32
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 26,
    "output_tokens": 6,
    "total_tokens": 32
  }
}

Multimodal messages

info

This feature is currently in preview. The message schema may change in future releases.

OpenAI supports interleaving images with text in input messages with their gpt-4-vision-preview. Here’s an example of how this looks:

import * as fs from "node:fs/promises";

import { ChatOpenAI } from "@langchain/openai";
import { HumanMessage } from "@langchain/core/messages";

const imageData2 = await fs.readFile("../../../../../examples/hotdog.jpg");
const llm2 = new ChatOpenAI({
  model: "gpt-4-vision-preview",
  maxTokens: 1024,
  apiKey: process.env.OPENAI_API_KEY,
});
const message2 = new HumanMessage({
  content: [
    {
      type: "text",
      text: "What's in this image?",
    },
    {
      type: "image_url",
      image_url: {
        url: `data:image/jpeg;base64,${imageData2.toString("base64")}`,
      },
    },
  ],
});

const res2 = await llm2.invoke([message2]);
console.log(res2);

AIMessage {
  "id": "chatcmpl-9rB59AKTPDrSHuTv0y7BNUcM0QDV2",
  "content": "The image shows a classic hot dog, consisting of a grilled or steamed sausage served in the slit of a partially sliced bun. The sausage appears to have grill marks, indicating it may have been cooked on a grill. This is a typical and popular snack or fast food item often enjoyed at sporting events, barbecues, and fairs.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 69,
      "promptTokens": 438,
      "totalTokens": 507
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 438,
    "output_tokens": 69,
    "total_tokens": 507
  }
}

const hostedImageMessage3 = new HumanMessage({
  content: [
    {
      type: "text",
      text: "What does this image say?",
    },
    {
      type: "image_url",
      image_url:
        "https://www.freecodecamp.org/news/content/images/2023/05/Screenshot-2023-05-29-at-5.40.38-PM.png",
    },
  ],
});
const res3 = await llm2.invoke([hostedImageMessage3]);
console.log(res3);

AIMessage {
  "id": "chatcmpl-9rB5EWz5AyOHg6UiFkt4HC8H4UZJu",
  "content": "The image contains text that reads \"LangChain\". Additionally, there is an illustration of a parrot on the left side and two interlinked rings on the right.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 33,
      "promptTokens": 778,
      "totalTokens": 811
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 778,
    "output_tokens": 33,
    "total_tokens": 811
  }
}

const lowDetailImage4 = new HumanMessage({
  content: [
    {
      type: "text",
      text: "Summarize the contents of this image.",
    },
    {
      type: "image_url",
      image_url: {
        url: "https://blog.langchain.dev/content/images/size/w1248/format/webp/2023/10/Screenshot-2023-10-03-at-4.55.29-PM.png",
        detail: "low",
      },
    },
  ],
});
const res4 = await llm2.invoke([lowDetailImage4]);
console.log(res4);

AIMessage {
  "id": "chatcmpl-9rB5IUbzvMo5nsOGYW3jvrQjaCiCg",
  "content": "The image shows a user interface of a digital service or platform called \"WebLangChain\" which appears to be powered by \"Tailify.\" There is a prompt that encourages users to \"Ask me anything about anything!\" Alongside this, there is a text input field labeled \"Ask anything...\" which also features some example questions or search queries such as \"what is langchain?\", \"history of mesopotamia\", \"how to build a discord bot\", \"leonardo dicaprio girlfriend\", \"fun gift ideas for software engineers\", \"how does a prism separate light\", and \"what bear is best\". The overall design is clean, with a dark background and a send button represented by a blue icon with a paper airplane, which typically symbolizes sending a message or submitting a query.",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 158,
      "promptTokens": 101,
      "totalTokens": 259
    },
    "finish_reason": "stop"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 101,
    "output_tokens": 158,
    "total_tokens": 259
  }
}

Tool calling

OpenAI chat models support calling multiple functions to get all required data to answer a question. Here’s an example how a conversation turn with this functionality might look:

import { ChatOpenAI } from "@langchain/openai";

// Bind function to the model as a tool
const llm5 = new ChatOpenAI({
  model: "gpt-3.5-turbo-1106",
  maxTokens: 128,
}).bind({
  tools: [
    {
      type: "function",
      function: {
        name: "get_current_weather",
        description: "Get the current weather in a given location",
        parameters: {
          type: "object",
          properties: {
            location: {
              type: "string",
              description: "The city and state, e.g. San Francisco, CA",
            },
            unit: { type: "string", enum: ["celsius", "fahrenheit"] },
          },
          required: ["location"],
        },
      },
    },
  ],
  tool_choice: "auto",
});

// Ask initial question that requires multiple tool calls
const res5 = await llm5.invoke([
  ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
]);
console.log(res5.tool_calls);

[
  {
    name: 'get_current_weather',
    args: { location: 'San Francisco', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_2ytmjITA18j3kLOzzjF5QSC4'
  },
  {
    name: 'get_current_weather',
    args: { location: 'Tokyo', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_3sU2dCNZ8e8A8wrYlYa7Xq0G'
  },
  {
    name: 'get_current_weather',
    args: { location: 'Paris', unit: 'celsius' },
    type: 'tool_call',
    id: 'call_Crmc0QG4x1VHRUyiwPsqzmQS'
  }
]

import { ToolMessage } from "@langchain/core/messages";

// Mocked out function, could be a database/API call in production
function getCurrentWeather(location: string, _unit?: string) {
  if (location.toLowerCase().includes("tokyo")) {
    return JSON.stringify({ location, temperature: "10", unit: "celsius" });
  } else if (location.toLowerCase().includes("san francisco")) {
    return JSON.stringify({
      location,
      temperature: "72",
      unit: "fahrenheit",
    });
  } else {
    return JSON.stringify({ location, temperature: "22", unit: "celsius" });
  }
}

// Format the results from calling the tool calls back to OpenAI as ToolMessages
const toolMessages5 = res5.additional_kwargs.tool_calls?.map((toolCall) => {
  const toolCallResult5 = getCurrentWeather(
    JSON.parse(toolCall.function.arguments).location
  );
  return new ToolMessage({
    tool_call_id: toolCall.id,
    name: toolCall.function.name,
    content: toolCallResult5,
  });
});

// Send the results back as the next step in the conversation
const finalResponse5 = await llm5.invoke([
  ["human", "What's the weather like in San Francisco, Tokyo, and Paris?"],
  res5,
  ...(toolMessages5 ?? []),
]);

console.log(finalResponse5);

AIMessage {
  "id": "chatcmpl-9rB5Sc3ERHpRymmAAsGS67zczVhAl",
  "content": "The current weather in:\n- San Francisco is 72°F\n- Tokyo is 10°C\n- Paris is 22°C",
  "additional_kwargs": {},
  "response_metadata": {
    "tokenUsage": {
      "completionTokens": 27,
      "promptTokens": 236,
      "totalTokens": 263
    },
    "finish_reason": "stop",
    "system_fingerprint": "fp_adbef9f124"
  },
  "tool_calls": [],
  "invalid_tool_calls": [],
  "usage_metadata": {
    "input_tokens": 236,
    "output_tokens": 27,
    "total_tokens": 263
  }
}

`.withStructuredOutput({ ... })`

You can also use the .withStructuredOutput({ ... }) method to coerce ChatOpenAI into returning a structured output.

The method allows for passing in either a Zod object, or a valid JSON schema (like what is returned from zodToJsonSchema).

Using the method is simple. Just define your LLM and call .withStructuredOutput({ ... }) on it, passing the desired schema.

Here is an example using a Zod schema and the functionCalling mode (default mode):

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";

const llm6 = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4-turbo-preview",
});

const calculatorSchema6 = z.object({
  operation: z.enum(["add", "subtract", "multiply", "divide"]),
  number1: z.number(),
  number2: z.number(),
});

const modelWithStructuredOutput6 = llm6.withStructuredOutput(calculatorSchema6);

const prompt6 = ChatPromptTemplate.fromMessages([
  ["system", "You are VERY bad at math and must always use a calculator."],
  ["human", "Please help me!! What is 2 + 2?"],
]);
const chain6 = prompt6.pipe(modelWithStructuredOutput6);
const result6 = await chain6.invoke({});
console.log(result6);

{ operation: 'add', number1: 2, number2: 2 }

You can also specify includeRaw to return the parsed and raw output in the result.

const includeRawModel6 = llm6.withStructuredOutput(calculatorSchema6, {
  name: "calculator",
  includeRaw: true,
});

const includeRawChain6 = prompt6.pipe(includeRawModel6);
const includeRawResult6 = await includeRawChain6.invoke({});
console.log(includeRawResult6);

{
  raw: AIMessage {
    "id": "chatcmpl-9rB5emIYRslBFrUIsC2368dXltljw",
    "content": "",
    "additional_kwargs": {
      "tool_calls": [
        {
          "id": "call_JaH5OB3KYvKF76TUOt6Lp8mu",
          "type": "function",
          "function": "[Object]"
        }
      ]
    },
    "response_metadata": {
      "tokenUsage": {
        "completionTokens": 15,
        "promptTokens": 93,
        "totalTokens": 108
      },
      "finish_reason": "stop"
    },
    "tool_calls": [
      {
        "name": "calculator",
        "args": {
          "number1": 2,
          "number2": 2,
          "operation": "add"
        },
        "type": "tool_call",
        "id": "call_JaH5OB3KYvKF76TUOt6Lp8mu"
      }
    ],
    "invalid_tool_calls": [],
    "usage_metadata": {
      "input_tokens": 93,
      "output_tokens": 15,
      "total_tokens": 108
    }
  },
  parsed: { operation: 'add', number1: 2, number2: 2 }
}

Additionally, you can pass in an OpenAI function definition or JSON schema directly:

info

If using jsonMode as the method you must include context in your prompt about the structured output you want. This must include the keyword: JSON.

import { ChatPromptTemplate } from "@langchain/core/prompts";
import { ChatOpenAI } from "@langchain/openai";

const llm7 = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4-turbo-preview",
});

const calculatorSchema7 = {
  type: "object",
  properties: {
    operation: {
      type: "string",
      enum: ["add", "subtract", "multiply", "divide"],
    },
    number1: { type: "number" },
    number2: { type: "number" },
  },
  required: ["operation", "number1", "number2"],
};

// Default mode is "functionCalling"
const modelWithStructuredOutput7 = llm7.withStructuredOutput(calculatorSchema7);

const prompt7 = ChatPromptTemplate.fromMessages([
  [
    "system",
    `You are VERY bad at math and must always use a calculator.
Respond with a JSON object containing three keys:
'operation': the type of operation to execute, either 'add', 'subtract', 'multiply' or 'divide',
'number1': the first number to operate on,
'number2': the second number to operate on.
`,
  ],
  ["human", "Please help me!! What is 2 + 2?"],
]);
const chain7 = prompt7.pipe(modelWithStructuredOutput7);
const result7 = await chain7.invoke({});
console.log(result7);

{ number1: 2, number2: 2, operation: 'add' }

You can also specify ‘includeRaw’ to return the parsed and raw output in the result, as well as a “name” field to give the LLM additional context as to what you are generating.

const includeRawModel7 = llm7.withStructuredOutput(calculatorSchema7, {
  name: "calculator",
  includeRaw: true,
  method: "jsonMode",
});

const includeRawChain7 = prompt7.pipe(includeRawModel7);
const includeRawResult7 = await includeRawChain7.invoke({});
console.log(includeRawResult7);

{
  raw: AIMessage {
    "id": "chatcmpl-9rB5lkylQMLSP9CQ4SaQB9zGw1rP1",
    "content": "{\n  \"operation\": \"add\",\n  \"number1\": 2,\n  \"number2\": 2\n}",
    "additional_kwargs": {},
    "response_metadata": {
      "tokenUsage": {
        "completionTokens": 25,
        "promptTokens": 91,
        "totalTokens": 116
      },
      "finish_reason": "stop"
    },
    "tool_calls": [],
    "invalid_tool_calls": [],
    "usage_metadata": {
      "input_tokens": 91,
      "output_tokens": 25,
      "total_tokens": 116
    }
  },
  parsed: { operation: 'add', number1: 2, number2: 2 }
}

Disabling parallel tool calls

If you have multiple tools bound to the model, but you’d only like for a single tool to be called at a time, you can pass the parallel_tool_calls call option to enable/disable this behavior. By default, parallel_tool_calls is set to true.

import { ChatOpenAI } from "@langchain/openai";
import { z } from "zod";
import { zodToJsonSchema } from "zod-to-json-schema";

const llm8 = new ChatOpenAI({
  temperature: 0,
  model: "gpt-4o",
});

// Define your tools
const calculatorSchema8 = z
  .object({
    operation: z.enum(["add", "subtract", "multiply", "divide"]),
    number1: z.number(),
    number2: z.number(),
  })
  .describe("A tool to perform basic arithmetic operations");
const weatherSchema8 = z
  .object({
    city: z.string(),
  })
  .describe("A tool to get the weather in a city");

// Bind tools to the model
const modelWithTools8 = llm8.bindTools([
  {
    type: "function",
    function: {
      name: "calculator",
      description: calculatorSchema8.description,
      parameters: zodToJsonSchema(calculatorSchema8),
    },
  },
  {
    type: "function",
    function: {
      name: "weather",
      description: weatherSchema8.description,
      parameters: zodToJsonSchema(weatherSchema8),
    },
  },
]);

// Invoke the model with `parallel_tool_calls` set to `true`
const response8 = await modelWithTools8.invoke(
  ["What is the weather in san francisco and what is 23716 times 27342?"],
  {
    parallel_tool_calls: true,
  }
);

We can see it called two tools:

console.log(response8.tool_calls);

[
  {
    name: 'weather',
    args: { city: 'san francisco' },
    type: 'tool_call',
    id: 'call_FyxazII0M0OgKMnk2UuXDhjv'
  },
  {
    name: 'calculator',
    args: { operation: 'multiply', number1: 23716, number2: 27342 },
    type: 'tool_call',
    id: 'call_raQz2ABUtVpbkruA2K6vBNYd'
  }
]

Invoke the model with parallel_tool_calls set to false

const response9 = await modelWithTools8.invoke(
  ["What is the weather in san francisco and what is 23716 times 27342?"],
  {
    parallel_tool_calls: false,
  }
);

We can see it called one tool

console.log(response9.tool_calls);

[
  {
    name: 'weather',
    args: { city: 'san francisco' },
    type: 'tool_call',
    id: 'call_xFbViRUVYj8BFnJIVedU7GVn'
  }
]

Custom URLs

You can customize the base URL the SDK sends requests to by passing a configuration parameter like this:

import { ChatOpenAI } from "@langchain/openai";

const llm10 = new ChatOpenAI({
  temperature: 0.9,
  configuration: {
    baseURL: "https://your_custom_url.com",
  },
});

const message10 = await llm10.invoke("Hi there!");

You can also pass other ClientOptions parameters accepted by the official SDK.

If you are hosting on Azure OpenAI, see the dedicated page instead.

Calling fine-tuned models

You can call fine-tuned OpenAI models by passing in your corresponding modelName parameter.

This generally takes the form of ft:{OPENAI_MODEL_NAME}:{ORG_NAME}::{MODEL_ID}. For example:

import { ChatOpenAI } from "@langchain/openai";

const llm11 = new ChatOpenAI({
  temperature: 0.9,
  model: "ft:gpt-3.5-turbo-0613:{ORG_NAME}::{MODEL_ID}",
});

const message11 = await llm11.invoke("Hi there!");

Generation metadata

If you need additional information like logprobs or token usage, these will be returned directly in the .invoke response.

tip

Requires @langchain/core version >=0.1.48.

import { ChatOpenAI } from "@langchain/openai";

// See https://cookbook.openai.com/examples/using_logprobs for details
const llm12 = new ChatOpenAI({
  logprobs: true,
  // topLogprobs: 5,
});

const responseMessage12 = await llm12.invoke("Hi there!");
console.dir(responseMessage12.response_metadata.logprobs, { depth: null });

{
  content: [
    {
      token: 'Hello',
      logprob: -0.0004585519,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.000049305523,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.000029517714,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.00073185476,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -9.0883464e-7,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.104538105,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -6.704273e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000052643223,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.00001247159,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

With callbacks

You can also use the callbacks system:

import { ChatOpenAI } from "@langchain/openai";

// See https://cookbook.openai.com/examples/using_logprobs for details
const llm13 = new ChatOpenAI({
  logprobs: true,
  // topLogprobs: 5,
});

const result13 = await llm13.invoke("Hi there!", {
  callbacks: [
    {
      handleLLMEnd(output) {
        console.dir(output.generations[0][0].generationInfo.logprobs, {
          depth: null,
        });
      },
    },
  ],
});

{
  content: [
    {
      token: 'Hello',
      logprob: -0.0005182436,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.000040246043,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.000035716304,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.0006764544,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -0.0000010280384,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.12827769,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -4.3202e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000059914648,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.000012352386,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

console.dir(result13.response_metadata.logprobs, { depth: null });

{
  content: [
    {
      token: 'Hello',
      logprob: -0.0005182436,
      bytes: [ 72, 101, 108, 108, 111 ],
      top_logprobs: []
    },
    {
      token: '!',
      logprob: -0.000040246043,
      bytes: [ 33 ],
      top_logprobs: []
    },
    {
      token: ' How',
      logprob: -0.000035716304,
      bytes: [ 32, 72, 111, 119 ],
      top_logprobs: []
    },
    {
      token: ' can',
      logprob: -0.0006764544,
      bytes: [ 32, 99, 97, 110 ],
      top_logprobs: []
    },
    {
      token: ' I',
      logprob: -0.0000010280384,
      bytes: [ 32, 73 ],
      top_logprobs: []
    },
    {
      token: ' assist',
      logprob: -0.12827769,
      bytes: [
         32,  97, 115,
        115, 105, 115,
        116
      ],
      top_logprobs: []
    },
    {
      token: ' you',
      logprob: -4.3202e-7,
      bytes: [ 32, 121, 111, 117 ],
      top_logprobs: []
    },
    {
      token: ' today',
      logprob: -0.000059914648,
      bytes: [ 32, 116, 111, 100, 97, 121 ],
      top_logprobs: []
    },
    {
      token: '?',
      logprob: -0.000012352386,
      bytes: [ 63 ],
      top_logprobs: []
    }
  ]
}

Streaming tokens

OpenAI supports streaming token counts via an opt-in call option. This can be set by passing { stream_options: { include_usage: true } }. Setting this call option will cause the model to return an additional chunk at the end of the stream, containing the token usage.

import type { AIMessageChunk } from "@langchain/core/messages";
import { ChatOpenAI } from "@langchain/openai";
import { concat } from "@langchain/core/utils/stream";

// Instantiate the model
const llm14 = new ChatOpenAI();

const response14 = await llm14.stream("Hello, how are you?", {
  // Pass the stream options
  stream_options: {
    include_usage: true,
  },
});

// Iterate over the response, only saving the last chunk
let finalResult14: AIMessageChunk | undefined;
for await (const chunk14 of response14) {
  finalResult14 = !finalResult14 ? chunk14 : concat(finalResult14, chunk14);
}

console.log(finalResult14?.usage_metadata);

{ input_tokens: 13, output_tokens: 33, total_tokens: 46 }

API reference

For detailed documentation of all ChatOpenAI features and configurations head to the API reference: https://api.js.langchain.com/classes/langchain_openai.ChatOpenAI.html

ChatOpenAI

Overview

Integration details

Model features

Setup

Credentials

Installation

Instantiation

Invocation

Chaining

Multimodal messages

Tool calling

`.withStructuredOutput({ ... })`

Disabling parallel tool calls

Custom URLs

Calling fine-tuned models

Generation metadata

With callbacks

Streaming tokens

API reference

Was this page helpful?

You can also leave detailed feedback on GitHub.

Overview​

Integration details​

Model features​

Setup​

Credentials​

Installation​

Instantiation​

Invocation​

Chaining​

Multimodal messages​

Tool calling​

.withStructuredOutput({ ... })​

Disabling parallel tool calls​

Custom URLs​

Calling fine-tuned models​

Generation metadata​

With callbacks​

Streaming tokens​

API reference​

Was this page helpful?

You can also leave detailed feedback on GitHub.

Overview

Integration details

Model features

Setup

Credentials

Installation

Instantiation

Invocation

Chaining

Multimodal messages

Tool calling

`.withStructuredOutput({ ... })`

Disabling parallel tool calls

Custom URLs

Calling fine-tuned models

Generation metadata

With callbacks

Streaming tokens

API reference