OpenAI builds its first custom AI chip

OpenAI has taken a major step toward controlling its own hardware stack with the reveal of Jalapeño, the company's first custom-designed AI chip. Co-developed with Broadcom and built specifically for large language model workloads, the chip is being positioned as a direct answer to the latency, cost, and efficiency pressures that come from running products like ChatGPT, Codex, and the API at scale. In benchmark results shared by OpenAI, Jalapeño delivered faster response times and better power efficiency than rival offerings, including current Nvidia GPUs.

Jalapeño is an application-specific integrated circuit, or ASIC, designed from the ground up for inference. Inference is the phase of AI processing in which a trained model applies what it has learned to new inputs and generates text, code, or other outputs. Training AI models is an enormous offline job that can require thousands of accelerators working for weeks, but inference is a continuous production workload. Every time a user asks ChatGPT a question, a chain of computational tasks begins. The speed of that chain determines how quickly the text appears on screen. OpenAI says Jalapeño was purpose-built for these language model workloads instead of being adapted from a general-purpose GPU or gaming silicon.

What do the new benchmarks show?

OpenAI tested Jalapeño using InferenceX, an independent benchmark from SemiAnalysis that measures how well a chip handles inference. InferenceX looks beyond raw theoretical peak performance and measures real-world conditions such as latency, throughput, and power consumption. The headline results are strong. Across three different AI models, Jalapeño delivered 1.5 to 1.9 times more AI work per watt of power. It also produced 1.7 to 3.6 times lower latency, which translates into noticeably faster response times for end users. For highly interactive tasks such as AI agents, the advantage was even larger, reaching up to 4.1 times better performance.

• 1.5 to 1.9 times more AI work per watt across three tested models
• 1.7 to 3.6 times lower latency
• Up to 4.1 times better performance for interactive agentic workloads

Most chips force a tradeoff between speed and efficiency. A processor can often be tuned to return answers quickly, but only by consuming more energy. Alternatively, it can be optimized for low power use, but users may wait longer for responses. OpenAI says Jalapeño manages to improve both at the same time by minimizing how much data needs to move between different parts of the system during processing. In modern AI inference, moving data between compute engines and memory is often the

Source: Digital Trends News