Spring Boot 3 + AI LLM Integration: The Complete Guide

技术架构AI agents in production

In 2026, Java Developers Must Embrace AI Large Language Models

Python trains models, Java applies them — this is the golden division of AI engineering in 2026.

One fact: 72% of backend systems in Fortune 500 companies run on the JVM, but 90% of AI application prototyping uses Python. The question is — once model training is done, who handles production traffic? The answer is Java.

Why Java is the Best Runtime for AI Applications

Dimension Python Java (Spring Boot)
Concurrency GIL-limited, single-threaded Multi-threaded + Virtual Threads (Loom), 10K+ concurrent
Enterprise Integration Requires lots of glue code Spring ecosystem, one-stop integration
Security & Compliance Dynamic typing, runtime errors Strong typing + compile-time checks
Operations Maturity Gunicorn/uWSGI Spring Boot Actuator + K8s
Team Skills AI researchers Enterprise backend engineers
Deployment Consistency Dependency hell Fat JAR, build once run anywhere

Three Scenarios for Java + AI

┌──────────────────────────────────────────────────────────┐
│                  Java + AI Application Landscape          │
├──────────────┬──────────────┬────────────────────────────┤
│ AI-Enhanced  │ AI-Native    │      AI Agent              │
│ Applications │ Applications │                            │
│              │              │                            │
│ Smart CS    │ ChatBot      │ Autonomous Agent           │
│ Doc Q&A     │ Code Copilot │ Planner→Executor→Evaluator │
│ Data Analyst│ RAG Knowledge│ Multi-tool Orchestration   │
│ Smart Recs  │ Content Gen  │ Workflow Automation        │
├──────────────┴──────────────┴────────────────────────────┤
│              Spring Boot 3 + Spring AI                   │
│    Unified Programming Model · Declarative Config        │
│    · Production-Grade Reliability                        │
└──────────────────────────────────────────────────────────┘

Spring AI vs LangChain4j: Framework Selection

The two dominant frameworks in the 2026 Java AI ecosystem, each with different emphases.

Core Positioning Comparison

Dimension Spring AI LangChain4j
Design Philosophy Spring-style, declarative LangChain port, chain-based
Core Team Spring official (VMware) Independent open-source community
Configuration application.yml + Bean Builder pattern + code config
Model Support OpenAI/Azure/Ollama/Tongyi OpenAI/Azure/Ollama/Tongyi/Zhipu
Vector Store PGVector/Chroma/Milvus PGVector/Chroma/Milvus/Weaviate
RAG Support Built-in ETL Pipeline Built-in RAG module
Function Calling Auto-register Spring Beans Manual @Tool method registration
Spring Integration Native, zero config Requires spring-boot-starter
Streaming Response Flux TokenStream
Community Activity ⭐⭐⭐⭐⭐ ⭐⭐⭐⭐
Learning Curve Zero barrier for Spring devs Must learn LangChain concepts
Version (2026) 1.0.0 GA 1.0.0 GA

Code Style Comparison

// Spring AI — Declarative, Spring-style
@Configuration
public class AiConfig {
    @Bean
    public ChatClient chatClient(ChatClient.Builder builder) {
        return builder
            .defaultSystem("You are a professional Java technical consultant")
            .defaultAdvisors(new SimpleLoggerAdvisor())
            .build();
    }
}

// Invocation: one-liner
String response = chatClient.prompt()
    .user("Explain Spring Boot 3 virtual threads")
    .call()
    .content();

// LangChain4j — Chain-based, Builder-style
ChatLanguageModel model = OpenAiChatModel.builder()
    .apiKey(System.getenv("OPENAI_API_KEY"))
    .modelName("gpt-4o")
    .temperature(0.7)
    .build();

// Invocation: explicit construction
Response<AiMessage> response = model.generate(
    SystemMessage.from("You are a professional Java technical consultant"),
    UserMessage.from("Explain Spring Boot 3 virtual threads")
);

Selection Recommendations

Scenario Recommendation Reason
Add AI to existing Spring project Spring AI Zero learning cost, native integration
Brand-new AI-native project Either Depends on team tech stack preference
Complex Agent orchestration LangChain4j Chain/Agent abstractions more mature
Enterprise compliance requirements Spring AI Actuator + Security integration
Rapid prototyping LangChain4j Builder pattern more intuitive

This article focuses on Spring AI as the primary framework, since its integration with Spring Boot 3 is the most natural and it is the mainstream choice for enterprise-grade Java AI applications in 2026.

Spring Boot 3 + Spring AI Quick Start

Step 1: Maven Dependencies

<?xml version="1.0" encoding="UTF-8"?>
<project>
    <parent>
        <groupId>org.springframework.boot</groupId>
        <artifactId>spring-boot-starter-parent</artifactId>
        <version>3.4.0</version>
    </parent>

    <properties>
        <java.version>21</java.version>
        <spring-ai.version>1.0.0</spring-ai.version>
    </properties>

    <dependencies>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-openai-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-pgvector-store-spring-boot-starter</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.ai</groupId>
            <artifactId>spring-ai-chat-memory-redis</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-web</artifactId>
        </dependency>
        <dependency>
            <groupId>org.springframework.boot</groupId>
            <artifactId>spring-boot-starter-actuator</artifactId>
        </dependency>
    </dependencies>

    <dependencyManagement>
        <dependencies>
            <dependency>
                <groupId>org.springframework.ai</groupId>
                <artifactId>spring-ai-bom</artifactId>
                <version>${spring-ai.version}</version>
                <type>pom</type>
                <scope>import</scope>
            </dependency>
        </dependencies>
    </dependencyManagement>
</project>

Step 2: YAML Configuration

spring:
  ai:
    openai:
      api-key: ${OPENAI_API_KEY}
      base-url: https://api.openai.com
      chat:
        options:
          model: gpt-4o
          temperature: 0.7
          max-tokens: 4096
          top-p: 0.9
      embedding:
        options:
          model: text-embedding-3-large
    vectorstore:
      pgvector:
        index-type: HNSW
        dimensions: 3072
        distance-type: COSINE
    chat:
      memory:
        redis:
          host: localhost
          port: 6379
          ttl: 3600

server:
  port: 8080

management:
  endpoints:
    web:
      exposure:
        include: health,info,metrics,prometheus

Step 3: ChatController

@RestController
@RequestMapping("/api/chat")
public class ChatController {

    private final ChatClient chatClient;

    public ChatController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @PostMapping
    public String chat(@RequestBody ChatRequest request) {
        return chatClient.prompt()
            .user(request.message())
            .call()
            .content();
    }

    @PostMapping(value = "/stream", produces = MediaType.TEXT_EVENT_STREAM_VALUE)
    public Flux<String> chatStream(@RequestBody ChatRequest request) {
        return chatClient.prompt()
            .user(request.message())
            .stream()
            .content();
    }

    @PostMapping("/system")
    public String chatWithSystem(
            @RequestBody ChatRequest request,
            @RequestParam String role) {
        return chatClient.prompt()
            .system(role)
            .user(request.message())
            .call()
            .content();
    }

    public record ChatRequest(String message) {}
}

Step 4: Start and Verify

export OPENAI_API_KEY=sk-xxxxx
mvn spring-boot:run

curl -X POST http://localhost:8080/api/chat \
  -H "Content-Type: application/json" \
  -d '{"message": "Write a quicksort in Java"}'

curl -X POST http://localhost:8080/api/chat/stream \
  -H "Content-Type: application/json" \
  -d '{"message": "Explain the JVM memory model"}'

┌──────────────────────────────────────────────────────┐
│         Spring Boot 3 + Spring AI Startup Flow        │
├──────────────────────────────────────────────────────┤
│                                                      │
│  main() → @SpringBootApplication                     │
│    ├── Auto-configure spring-ai-auto-configuration   │
│    ├── Register OpenAiChatModel Bean                 │
│    ├── Register OpenAiEmbeddingModel Bean            │
│    ├── Register PgVectorStore Bean                   │
│    ├── Register ChatClient Bean                      │
│    └── Register ChatMemory Bean (Redis)              │
│                                                      │
│  ChatClient.prompt()                                 │
│    ├── .user() → Build UserMessage                   │
│    ├── .system() → Build SystemMessage               │
│    ├── .advisors() → Inject Advisor chain            │
│    ├── .call() → Synchronous call → String           │
│    └── .stream() → Streaming call → Flux<String>     │
│                                                      │
└──────────────────────────────────────────────────────┘

Enterprise Chat: Multi-Turn Dialogue with Redis Persistent Memory

An AI without memory is like a goldfish — every conversation is brand new, unable to understand context. Enterprise chat applications must have persistent memory.

Conversation Memory Architecture

┌──────────────────────────────────────────────────────────┐
│              Multi-Turn Dialogue Memory Architecture      │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  User Message ──→ ChatMemoryAdvisor ──→ ChatClient       │
│                  │                      │                │
│                  ▼                      ▼                │
│            ┌──────────┐          ┌──────────┐          │
│            │  Redis   │          │ OpenAI   │          │
│            │ ChatStore│          │  API     │          │
│            └──────────┘          └──────────┘          │
│                  ▲                      │                │
│                  │                      ▼                │
│            Write Back ←── AI Response ←── Model Inference│
│                                                          │
│  Memory Structure:                                       │
│  conversation:{userId} → [                               │
│    {role: "system", content: "..."},                     │
│    {role: "user", content: "..."},                       │
│    {role: "assistant", content: "..."},                  │
│    ...                                                   │
│  ]                                                       │
└──────────────────────────────────────────────────────────┘

Configure ChatMemory

@Configuration
public class ChatMemoryConfig {

    @Bean
    public ChatClient chatClientWithMemory(
            ChatClient.Builder builder,
            ChatMemoryRepository memoryRepository) {

        ChatMemory chatMemory = new RedisChatMemory(memoryRepository);

        return builder
            .defaultSystem("""
                You are the AI assistant for ToolKit website,
                specializing in Java and Spring technologies.
                Provide professional, accurate answers with code examples.
                If unsure, honestly say so.
                """)
            .defaultAdvisors(
                MessageChatMemoryAdvisor.builder(chatMemory)
                    .conversationIdExpression("userId")
                    .maxMessages(20)
                    .build(),
                new SimpleLoggerAdvisor()
            )
            .build();
    }
}

ChatController with Memory

@RestController
@RequestMapping("/api/chat")
public class EnterpriseChatController {

    private final ChatClient chatClient;

    public EnterpriseChatController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @PostMapping("/conversation")
    public ChatResponse conversation(
            @RequestBody ConversationRequest request,
            @RequestHeader("X-User-Id") String userId) {

        String content = chatClient.prompt()
            .user(request.message())
            .advisors(a -> a.param("userId", userId))
            .call()
            .content();

        return new ChatResponse(content, userId, Instant.now());
    }

    @PostMapping("/conversation/stream")
    public Flux<ServerSentEvent<String>> conversationStream(
            @RequestBody ConversationRequest request,
            @RequestHeader("X-User-Id") String userId) {

        return chatClient.prompt()
            .user(request.message())
            .advisors(a -> a.param("userId", userId))
            .stream()
            .content()
            .map(chunk -> ServerSentEvent.<String>builder()
                .data(chunk).build());
    }

    @DeleteMapping("/conversation/{userId}")
    public ResponseEntity<Void> clearMemory(@PathVariable String userId) {
        return ResponseEntity.noContent().build();
    }

    public record ConversationRequest(String message) {}
    public record ChatResponse(String content, String userId, Instant timestamp) {}
}

Redis Memory Store Implementation

@Component
public class RedisChatMemoryRepository implements ChatMemoryRepository {

    private final StringRedisTemplate redisTemplate;
    private final ObjectMapper objectMapper;
    private static final String KEY_PREFIX = "chat:memory:";
    private static final Duration TTL = Duration.ofHours(2);

    public RedisChatMemoryRepository(
            StringRedisTemplate redisTemplate,
            ObjectMapper objectMapper) {
        this.redisTemplate = redisTemplate;
        this.objectMapper = objectMapper;
    }

    @Override
    public List<Message> findByConversationId(String conversationId) {
        String json = redisTemplate.opsForValue().get(KEY_PREFIX + conversationId);
        if (json == null) return new ArrayList<>();
        try {
            return objectMapper.readValue(json,
                new TypeReference<List<Message>>() {});
        } catch (JsonProcessingException e) {
            throw new ChatMemoryException("Failed to read memory", e);
        }
    }

    @Override
    public void saveAll(String conversationId, List<Message> messages) {
        try {
            String json = objectMapper.writeValueAsString(messages);
            redisTemplate.opsForValue()
                .set(KEY_PREFIX + conversationId, json, TTL);
        } catch (JsonProcessingException e) {
            throw new ChatMemoryException("Failed to save memory", e);
        }
    }
}

Memory Strategy Comparison

Strategy Implementation Pros Cons Use Case
Window Memory Keep last N turns Simple, token-controlled Loses early context General chat
Summary Memory Compress old turns to summary Preserves global semantics Summary may lose details Long conversations
Hybrid Memory Summary + last N turns Both global and detail Complex implementation Enterprise CS
Vector Memory Semantic retrieval of relevant chunks Unlimited context Retrieval latency Knowledge-intensive

RAG in Practice: Document ETL Pipeline + Vector Retrieval + Context Injection

RAG (Retrieval-Augmented Generation) is the core technology that gives LLMs access to enterprise private knowledge.

Complete RAG Architecture

┌──────────────────────────────────────────────────────────────┐
│                    Complete RAG Pipeline                       │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  ┌─────────┐    ┌──────────┐    ┌──────────┐               │
│  │  Doc     │───→│  ETL     │───→│  Vector  │               │
│  │  Source  │    │ Pipeline │    │  Store   │               │
│  │ PDF/MD  │    │          │    │ PGVector │               │
│  └─────────┘    └──────────┘    └──────────┘               │
│                      │              ▲                       │
│                      ▼              │                       │
│              ┌──────────────┐      │                       │
│              │  Embedding   │──────┘                       │
│              │  text-embed-3│                               │
│              └──────────────┘                               │
│                                                              │
│  ┌─────────┐    ┌──────────┐    ┌──────────┐               │
│  │  User    │───→│ Retrieve │───→│ Generate │               │
│  │  Query   │    │ Top-K    │    │ Ctx + Q  │               │
│  └─────────┘    └──────────┘    └──────────┘               │
│                      │              │                       │
│                      ▼              ▼                       │
│              ┌──────────────┐  ┌──────────────┐           │
│              │  RRF Rerank  │  │  Answer+Cite │           │
│              └──────────────┘  └──────────────┘           │
└──────────────────────────────────────────────────────────────┘

Document ETL Pipeline

@Service
public class DocumentEtlService {

    private final VectorStore vectorStore;
    private final DocumentReader documentReader;
    private final DocumentTransformer documentTransformer;

    public DocumentEtlService(VectorStore vectorStore,
            DocumentReader documentReader,
            DocumentTransformer documentTransformer) {
        this.vectorStore = vectorStore;
        this.documentReader = documentReader;
        this.documentTransformer = documentTransformer;
    }

    public void ingestDocuments(String directoryPath) {
        List<Document> documents = documentReader.read(directoryPath);
        List<Document> chunks = documentTransformer.apply(documents);
        vectorStore.add(chunks);
        log.info("Ingested {} documents, {} chunks into vector store",
            documents.size(), chunks.size());
    }
}

Document Reader

@Component
public class SmartDocumentReader implements DocumentReader {

    @Override
    public List<Document> read(String path) {
        return switch (getFileExtension(path)) {
            case "pdf" -> readPdf(path);
            case "md"  -> readMarkdown(path);
            case "docx" -> readDocx(path);
            default -> throw new UnsupportedDocumentException(path);
        };
    }

    private List<Document> readPdf(String path) {
        var reader = new PagePdfDocumentReader(path,
            PdfDocumentReaderConfig.builder().pagesPerDocument(1).build());
        return reader.get();
    }

    private List<Document> readMarkdown(String path) {
        var reader = new MarkdownDocumentReader(path,
            MarkdownDocumentReaderConfig.builder().includeCodeBlocks(true).build());
        return reader.get();
    }
}

Document Chunking and Metadata Enrichment

@Component
public class SmartDocumentTransformer implements DocumentTransformer {

    private final TokenTextSplitter splitter;

    @Override
    public List<Document> apply(List<Document> documents) {
        return documents.stream()
            .flatMap(doc -> splitter.split(doc).stream())
            .peek(this::enrichMetadata)
            .toList();
    }

    private void enrichMetadata(Document chunk) {
        Map<String, Object> metadata = chunk.getMetadata();
        metadata.put("chunkId", UUID.randomUUID().toString());
        metadata.put("createdAt", Instant.now().toString());
        metadata.put("tokenCount", estimateTokenCount(chunk.getContent()));
        metadata.put("version", "2026-Q2");
    }

    private int estimateTokenCount(String text) {
        return text.length() / 4;
    }
}

RAG Retrieval Service

@Service
public class RagService {

    private final ChatClient chatClient;
    private final VectorStore vectorStore;

    public RagService(ChatClient chatClient, VectorStore vectorStore) {
        this.chatClient = chatClient;
        this.vectorStore = vectorStore;
    }

    public String query(String question) {
        List<Document> relevantDocs = vectorStore.similaritySearch(
            SearchRequest.builder()
                .query(question).topK(5).similarityThreshold(0.7).build());

        String context = relevantDocs.stream()
            .map(doc -> "[Source: %s]\n%s".formatted(
                doc.getMetadata().get("source"), doc.getContent()))
            .collect(Collectors.joining("\n---\n"));

        return chatClient.prompt()
            .system("""
                Answer based on the following reference documents.
                If documents lack relevant info, say "Cannot answer based on available documents."
                Cite your sources.
                Reference documents: {context}
                """)
            .user(question).call().content();
    }
}

RAG Performance Optimization Checklist

Optimization Method Effect
Chunk Size 512-1024 tokens, overlap 64-128 Balance semantic integrity and retrieval precision
Query Expansion HyDE + synonym expansion Recall improvement 15-30%
Hybrid Retrieval Vector + BM25 keyword, RRF fusion Exact match + semantic match complement
Reranking Cross-Encoder / Cohere Rerank Top-5 precision improvement 20%
Metadata Filtering Filter by department/version/date Reduce irrelevant document noise
Caching Cache results for similar queries Duplicate query latency reduction 90%

Function Calling: Let LLMs Invoke Java Methods

Function Calling is the bridge between LLMs and the external world — the model decides "when to call", you define "what to call".

Function Calling Workflow

┌──────────────────────────────────────────────────────────┐
│              Function Calling Workflow                     │
├──────────────────────────────────────────────────────────┤
│                                                          │
│  User: "Check logistics of order ORD-20260601"           │
│    │                                                     │
│    ▼                                                     │
│  ┌──────────┐                                            │
│  │   LLM    │ → Analyze intent → Select: queryLogistics  │
│  └──────────┘   Args: {orderId: "ORD-20260601"}         │
│    │                                                     │
│    ▼                                                     │
│  ┌──────────┐                                            │
│  │ Java     │ → Call queryLogistics("ORD-20260601")      │
│  │ Method   │   Return: {status: "In Transit", ...}      │
│  └──────────┘                                            │
│    │                                                     │
│    ▼                                                     │
│  ┌──────────┐                                            │
│  │   LLM    │ → Generate natural language from result    │
│  └──────────┘                                            │
│                                                          │
└──────────────────────────────────────────────────────────┘

Order Query Functions

@Configuration
public class OrderFunctions {

    @Bean
    @Description("Query order details by order ID")
    public Function<OrderQuery, OrderInfo> queryOrder(OrderService orderService) {
        return query -> orderService.getOrderInfo(query.orderId());
    }

    @Bean
    @Description("Query logistics status by order ID")
    public Function<LogisticsQuery, LogisticsInfo> queryLogistics(
            LogisticsService logisticsService) {
        return query -> logisticsService.getLogisticsInfo(query.orderId());
    }

    @Bean
    @Description("Submit a refund request for an order")
    public Function<RefundRequest, RefundResult> requestRefund(
            RefundService refundService) {
        return request -> refundService.processRefund(
            request.orderId(), request.reason());
    }

    public record OrderQuery(String orderId) {}
    public record LogisticsQuery(String orderId) {}
    public record RefundRequest(String orderId, String reason) {}
    public record OrderInfo(String orderId, String productName,
        BigDecimal amount, String status, LocalDateTime orderTime) {}
    public record LogisticsInfo(String orderId, String status,
        String currentLocation, LocalDateTime estimatedArrival) {}
    public record RefundResult(String refundId, String orderId,
        BigDecimal refundAmount, String status) {}
}

Function Calling Controller

@RestController
@RequestMapping("/api/assistant")
public class AiAssistantController {

    private final ChatClient chatClient;

    public AiAssistantController(ChatClient chatClient) {
        this.chatClient = chatClient;
    }

    @PostMapping
    public String assist(@RequestBody AssistRequest request) {
        return chatClient.prompt()
            .system("""
                You are an e-commerce smart customer service assistant. You can:
                1. Query order details
                2. Query logistics status
                3. Process refund requests
                Be friendly, professional, and include specific information.
                """)
            .user(request.message())
            .functions("queryOrder", "queryLogistics", "requestRefund")
            .call()
            .content();
    }

    public record AssistRequest(String message) {}
}

Example Dialogue

User: "Where is my order ORD-20260601?"

AI Internal Flow:
1. Intent recognition → Query logistics
2. Call queryLogistics({orderId: "ORD-20260601"})
3. Returns: {status: "In Transit", currentLocation: "Hangzhou Hub", ...}
4. Generate answer:

AI: "Your order ORD-20260601 is currently in transit at Hangzhou Hub,
     estimated delivery June 7, 2026."

From Chat to Agent: Building a Java AI Agent

Chat is "you ask, I answer". Agent is "you set a goal, I accomplish it autonomously".

Agent Core Loop

┌──────────────────────────────────────────────────────────┐
│                  AI Agent Core Loop                        │
│                                                          │
│   ┌──────────┐     ┌──────────┐     ┌──────────┐        │
│   │ Planner  │────→│ Executor │────→│Evaluator │        │
│   │          │     │          │     │          │        │
│   └──────────┘     └──────────┘     └──────────┘        │
│        ▲                                   │             │
│        │          Not satisfied             │             │
│        └───────────────────────────────────┘             │
│                                                          │
│   Planner:  Decompose goal into executable step list     │
│   Executor: Execute step by step, each step may call tools│
│   Evaluator: Check if result satisfies goal, decide retry │
│                                                          │
│   Termination: Evaluation passed / Max retries / User OK │
└──────────────────────────────────────────────────────────┘

Agent Core Implementation

@Service
public class AiAgentService {

    private final ChatClient chatClient;
    private final List<AgentTool> tools;

    public AiAgentService(ChatClient chatClient, List<AgentTool> tools) {
        this.chatClient = chatClient;
        this.tools = tools;
    }

    public AgentResult execute(String goal) {
        int maxIterations = 5;
        Plan plan = plan(goal);

        List<StepResult> results = new ArrayList<>();
        for (Step step : plan.steps()) {
            results.add(executeStep(step, results));
        }

        Evaluation evaluation = evaluate(goal, results);
        int iteration = 1;

        while (!evaluation.satisfied() && iteration < maxIterations) {
            plan = replan(goal, results, evaluation.feedback());
            results.clear();
            for (Step step : plan.steps()) {
                results.add(executeStep(step, results));
            }
            evaluation = evaluate(goal, results);
            iteration++;
        }

        return new AgentResult(results, evaluation, iteration);
    }

    private Plan plan(String goal) {
        String planJson = chatClient.prompt()
            .system("Decompose the goal into executable steps. " +
                "Available tools: %s. Output JSON array.".formatted(getToolDescriptions()))
            .user("Goal: " + goal)
            .call().content();
        return parsePlan(planJson);
    }

    private StepResult executeStep(Step step, List<StepResult> previousResults) {
        AgentTool tool = findTool(step.tool());
        Object result = tool.execute(step.params());
        return new StepResult(step, result, Instant.now());
    }

    private Evaluation evaluate(String goal, List<StepResult> results) {
        String evalJson = chatClient.prompt()
            .system("Evaluate if results satisfy the goal. " +
                "Output: {\"satisfied\":boolean,\"score\":0-100,\"feedback\":\"...\"}")
            .user("Goal: %s\nResults: %s".formatted(goal, formatResults(results)))
            .call().content();
        return parseEvaluation(evalJson);
    }

    public record Plan(List<Step> steps) {}
    public record Step(String action, String tool, Map<String, Object> params) {}
    public record StepResult(Step step, Object result, Instant timestamp) {}
    public record Evaluation(boolean satisfied, int score, String feedback) {}
    public record AgentResult(List<StepResult> results, Evaluation evaluation, int iterations) {}
}

Agent Tool Registration

public interface AgentTool {
    String name();
    String description();
    Object execute(Map<String, Object> params);
}

@Component
public class DatabaseQueryTool implements AgentTool {
    private final JdbcTemplate jdbcTemplate;

    @Override
    public String name() { return "database_query"; }
    @Override
    public String description() { return "Execute SQL query, SELECT only"; }
    @Override
    public Object execute(Map<String, Object> params) {
        String sql = (String) params.get("sql");
        if (!sql.trim().toUpperCase().startsWith("SELECT"))
            throw new SecurityException("Only SELECT queries are allowed");
        return jdbcTemplate.queryForList(sql);
    }
}

@Component
public class HttpApiTool implements AgentTool {
    private final RestClient restClient;

    @Override
    public String name() { return "http_api"; }
    @Override
    public String description() { return "Call external HTTP API"; }
    @Override
    public Object execute(Map<String, Object> params) {
        String url = (String) params.get("url");
        return restClient.get().uri(url).retrieve().body(String.class);
    }
}

Multi-Model Routing and Fallback Strategy

Production environments cannot put all eggs in one basket — when GPT-4o goes down, Qwen steps up.

Multi-Model Routing Architecture

┌──────────────────────────────────────────────────────────────┐
│                   Multi-Model Routing Architecture            │
├──────────────────────────────────────────────────────────────┤
│                                                              │
│  Request ──→ ModelRouter ──→ ┌─────────┐                    │
│                              │ Routing │                    │
│                              │ Strategy│                    │
│                              └────┬────┘                    │
│                 ┌──────────────────┼──────────────┐        │
│                 ▼                  ▼              ▼        │
│          ┌──────────┐      ┌──────────┐  ┌──────────┐    │
│          │  GPT-4o  │      │  Qwen    │  │ DeepSeek │    │
│          │ (Premium)│      │ (China)  │  │ (Budget) │    │
│          └──────────┘      └──────────┘  └──────────┘    │
│                 │                  │              │        │
│                 ▼                  ▼              ▼        │
│          ┌─────────────────────────────────────────┐      │
│          │          Fallback Chain                  │      │
│          │  GPT-4o → Qwen → DeepSeek → Local Ollama│     │
│          └─────────────────────────────────────────┘      │
│                                                              │
└──────────────────────────────────────────────────────────────┘

Routing Strategy Implementation

@Service
public class ModelRouterService {

    private final Map<String, ChatModel> models;
    private final CircuitBreakerRegistry breakerRegistry;

    public ModelRouterService(
            @Qualifier("openai") ChatModel openai,
            @Qualifier("tongyi") ChatModel tongyi,
            @Qualifier("deepseek") ChatModel deepseek,
            @Qualifier("ollama") ChatModel ollama,
            CircuitBreakerRegistry breakerRegistry) {
        this.models = Map.of(
            "gpt-4o", openai, "qwen-max", tongyi,
            "deepseek-v3", deepseek, "llama3", ollama);
        this.breakerRegistry = breakerRegistry;
    }

    public String chat(String prompt, RoutingStrategy strategy) {
        List<String> modelChain = strategy.resolveChain(prompt);

        for (String modelName : modelChain) {
            try {
                CircuitBreaker breaker = breakerRegistry.circuitBreaker(modelName);
                ChatModel model = models.get(modelName);
                String result = breaker.executeSupplier(() ->
                    model.call(new Prompt(prompt))
                        .getResult().getOutput().getText());
                log.info("Model {} succeeded", modelName);
                return result;
            } catch (CallNotPermittedException e) {
                log.warn("Model {} circuit breaker open, trying next", modelName);
            } catch (Exception e) {
                log.error("Model {} failed: {}", modelName, e.getMessage());
            }
        }
        throw new AllModelsFailedException("All models in chain failed");
    }
}

Smart Routing Strategy

@Component
public class SmartRoutingStrategy implements RoutingStrategy {

    @Override
    public List<String> resolveChain(String prompt) {
        Complexity complexity = analyzeComplexity(prompt);
        return switch (complexity) {
            case HIGH   -> List.of("gpt-4o", "qwen-max", "deepseek-v3", "llama3");
            case MEDIUM -> List.of("qwen-max", "deepseek-v3", "gpt-4o");
            case LOW    -> List.of("deepseek-v3", "qwen-max", "llama3");
        };
    }

    private Complexity analyzeComplexity(String prompt) {
        if (prompt.length() > 500 || containsCodeRequest(prompt)) return Complexity.HIGH;
        if (prompt.length() > 100 || containsReasoning(prompt)) return Complexity.MEDIUM;
        return Complexity.LOW;
    }

    enum Complexity { HIGH, MEDIUM, LOW }
}

Multi-Model Cost Comparison

Model Input Price (/1M tokens) Output Price (/1M tokens) Quality Latency Use Case
GPT-4o $2.50 $10.00 ⭐⭐⭐⭐⭐ 1-3s Complex reasoning, code generation
Qwen-Max $1.10 $4.40 ⭐⭐⭐⭐ 0.5-2s Chinese scenarios, compliance
DeepSeek-V3 $0.14 $0.28 ⭐⭐⭐⭐ 0.3-1s Daily chat, high concurrency
Llama3 (local) Free Free ⭐⭐⭐ 2-5s Privacy-sensitive, offline

Production-Grade Deployment and Performance Tuning

The gap between Demo and Production lies in these details.

Docker Deployment

FROM eclipse-temurin:21-jre-alpine
WORKDIR /app
COPY target/app.jar app.jar

ENV JAVA_OPTS="-XX:+UseZGC \
  -XX:+ZGenerational \
  -XX:MaxRAMPercentage=75.0 \
  -XX:+EnableVirtualThreads \
  -Dspring.profiles.active=prod"

EXPOSE 8080
ENTRYPOINT ["sh", "-c", "java $JAVA_OPTS -jar app.jar"]

services:
  app:
    build: .
    ports: ["8080:8080"]
    environment:
      OPENAI_API_KEY: ${OPENAI_API_KEY}
    depends_on:
      redis: { condition: service_healthy }
      postgres: { condition: service_healthy }
    deploy:
      resources:
        limits: { memory: 1G, cpus: "2.0" }
    healthcheck:
      test: ["CMD", "wget", "-q", "--spider", "http://localhost:8080/actuator/health"]
      interval: 10s

  redis:
    image: redis:7-alpine
    ports: ["6379:6379"]
    healthcheck:
      test: ["CMD", "redis-cli", "ping"]

  postgres:
    image: pgvector/pgvector:pg16
    ports: ["5432:5432"]
    environment:
      POSTGRES_DB: ai_vectors
      POSTGRES_PASSWORD: ${PG_PASSWORD}
    volumes:
      - pgdata:/var/lib/postgresql/data

volumes:
  pgdata:

Rate Limiting

@Configuration
public class RateLimitConfig {
    @Bean
    public RateLimiter aiApiRateLimiter() {
        return RateLimiter.builder()
            .name("ai-api")
            .limitForPeriod(50)
            .limitRefreshPeriod(Duration.ofSeconds(1))
            .timeoutDuration(Duration.ofSeconds(5))
            .build();
    }
}

Monitoring Metrics

@Component
public class AiMetrics {
    private final MeterRegistry registry;

    public AiMetrics(MeterRegistry registry) { this.registry = registry; }

    public void recordApiCall(String model, boolean success, long latencyMs) {
        registry.counter("ai.api.calls",
            "model", model, "status", success ? "success" : "failure").increment();
        registry.timer("ai.api.latency", "model", model)
            .record(latencyMs, TimeUnit.MILLISECONDS);
    }

    public void recordTokenUsage(String model, int promptTokens, int completionTokens) {
        registry.counter("ai.tokens.prompt", "model", model).increment(promptTokens);
        registry.counter("ai.tokens.completion", "model", model).increment(completionTokens);
    }
}

Production Performance Checklist

Category Check Item Target Tool
Latency P95 API response time < 3s Actuator + Grafana
Latency Streaming first token time < 500ms Custom Metrics
Throughput Concurrent request capacity > 100 QPS JMeter/k6
Reliability Circuit breaker trigger rate < 5% Resilience4j
Reliability Fallback success rate > 99% Custom Metrics
Cost Daily token consumption < Budget Token counter
Cost Model routing hit rate High complexity → GPT-4o >80% Router Metrics
Security API Key rotation Every 90 days Vault/KMS
Security Sensitive info filtering 100% interception Input/Output Guard
Ops Zero-downtime rolling deploy 0 errors K8s ReadinessProbe
Ops Hot config reload No restart needed Spring Cloud Config

Summary and Architecture Overview

Full Architecture Diagram

┌──────────────────────────────────────────────────────────────────┐
│                  Spring Boot 3 + AI Full Architecture             │
├──────────────────────────────────────────────────────────────────┤
│                                                                  │
│  ┌────────────────────────────────────────────────────────┐     │
│  │                    API Gateway / Load Balancer          │     │
│  └──────────────────────┬─────────────────────────────────┘     │
│                         │                                       │
│  ┌──────────────────────▼─────────────────────────────────┐     │
│  │              Spring Boot 3 Application                  │     │
│  │                                                        │     │
│  │  ┌──────────┐  ┌──────────┐  ┌──────────┐            │     │
│  │  │ Chat     │  │ RAG      │  │ Agent    │            │     │
│  │  │ Controller│  │ Service  │  │ Service  │            │     │
│  │  └────┬─────┘  └────┬─────┘  └────┬─────┘            │     │
│  │       │             │             │                    │     │
│  │  ┌────▼─────────────▼─────────────▼─────┐            │     │
│  │  │         ChatClient (Spring AI)       │            │     │
│  │  │  ┌─────────────────────────────┐    │            │     │
│  │  │  │      Advisor Chain          │    │            │     │
│  │  │  │  Memory → RAG → Function   │    │            │     │
│  │  │  │  → Guard → Logging         │    │            │     │
│  │  │  └─────────────────────────────┘    │            │     │
│  │  └──────────────┬──────────────────────┘            │     │
│  │                  │                                    │     │
│  │  ┌──────────────▼──────────────────────┐            │     │
│  │  │         ModelRouter                 │            │     │
│  │  │  GPT-4o │ Qwen │ DeepSeek │ Ollama │            │     │
│  │  └─────────────────────────────────────┘            │     │
│  └──────────────────────────────────────────────────────┘     │
│                         │                                       │
│  ┌──────────────────────▼─────────────────────────────────┐     │
│  │              Infrastructure Layer                       │     │
│  │  Redis(Memory) │ PGVector(Vectors) │ MySQL(Biz) │ K8s │     │
│  └────────────────────────────────────────────────────────┘     │
│                                                                  │
└──────────────────────────────────────────────────────────────────┘

Key Takeaways

  1. Java is the best production runtime for AI — Concurrency, security, and operations in one, Python for training, Java for serving
  2. Spring AI is the framework of choice for 2026 — Spring official, declarative config, zero learning curve
  3. Multi-turn dialogue needs persistent memory — Redis ChatMemory, window/summary/hybrid strategies by scenario
  4. RAG is the cornerstone of enterprise AI — ETL Pipeline + vector retrieval + context injection, all indispensable
  5. Function Calling connects models to the world — Model decides when to call, you define what to call
  6. Agent = Planner + Executor + Evaluator — From passive response to proactive execution
  7. Multi-model routing is a production standard — GPT-4o + Qwen + DeepSeek, Fallback chain ensures availability
  8. Production deployment is non-negotiable — Docker + circuit breaker + rate limiting + monitoring, all required

Spring Boot 3 + AI LLMs is not "Java chasing AI" — it is "AI finally finding its most reliable production runtime." In 2026, the AI era for Java developers has arrived.

Try these browser-local tools — no sign-up required →

#Spring Boot#Spring AI#LangChain4j#AI大模型#RAG#Function Calling#Java Agent