Rust Tokio Graceful Shutdown: 5 Core Patterns for Production-Ready Async Service Lifecycle
编程语言David Liu
In the cloud-native landscape of 2026, a Rust async service is no longer a "start and forget" program. Kubernetes rolling updates, SIGTERM signals from container orchestrators, graceful draining of long-lived connections—every环节 demands precise control over the Tokio runtime lifecycle. A brutal ctrl+c can lead to data loss, interrupted requests, or even cascading failures. Today, let's dive deep into 5 core patterns for Rust Tokio graceful shutdown, from signal capture to task cancellation, from connection draining to multi-service coordination.
Core Concepts at a Glance
| Concept | Description | Key Types/Functions |
|---|---|---|
| Signal Capture | Listen for OS termination signals to trigger shutdown | tokio::signal, ctrl_c() |
| Cancellation Token | Cooperative task cancellation mechanism | tokio_util::sync::CancellationToken |
| Task Management | Track and wait for all spawned tasks | JoinHandle, JoinSet |
| Connection Draining | Stop accepting new connections, wait for existing ones | graceful_shutdown(), timeout control |
| Shutdown Manager | Orchestrate multi-component shutdown order | Custom ShutdownManager |
Problem Analysis: 5 Pain Points
- Signal Loss: Only listening for signals in main, subtasks can't detect shutdown events, causing background tasks to run forever
- Task Leaks: Spawned 100 tasks, only waited for 3 during shutdown, the other 97 become orphans
- Hard Connection Drops: Exiting immediately on SIGTERM, in-flight HTTP requests get truncated, clients receive connection reset
- Chaotic Shutdown Order: Database pool closes first, but tasks are still writing data, causing panics or data loss
- Timeout Runaway: A stuck task never exits, blocking the entire shutdown process, Kubernetes sends SIGKILL after timeout
Pattern 1: Signal Capture with tokio::signal and CancellationToken
This is the starting point for graceful shutdown—you must first "hear" the shutdown signal, then broadcast it to all tasks.
use tokio::signal;
use tokio_util::sync::CancellationToken;
use tokio::time::{sleep, Duration};
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let token = CancellationToken::new();
// Clone the cancellation token for each task
let t1 = token.clone();
let handle1 = tokio::spawn(async move {
tokio::select! {
_ = t1.cancelled() => {
println!("Task 1 received cancellation signal, starting cleanup...");
}
_ = async {
loop {
println!("Task 1 working...");
sleep(Duration::from_secs(1)).await;
}
} => {}
}
});
let t2 = token.clone();
let handle2 = tokio::spawn(async move {
tokio::select! {
_ = t2.cancelled() => {
println!("Task 2 received cancellation signal, starting cleanup...");
}
_ = async {
loop {
println!("Task 2 working...");
sleep(Duration::from_secs(2)).await;
}
} => {}
}
});
// Listen for multiple signal types
println!("Service started, press Ctrl+C or send SIGTERM to trigger graceful shutdown...");
tokio::select! {
_ = signal::ctrl_c() => {
println!("\nReceived Ctrl+C signal");
}
_ = signal::unix::signal(signal::unix::SignalKind::terminate())?.recv() => {
println!("Received SIGTERM signal");
}
}
// Broadcast cancellation
token.cancel();
// Wait for all tasks to complete
let _ = handle1.await;
let _ = handle2.await;
println!("All tasks have gracefully exited");
Ok(())
}
Key Takeaways:
CancellationTokenis cooperative cancellation, not forced termination—tasks must actively checkcancelled()- Use
tokio::select!to let tasks listen for both cancellation signals and normal work token.clone()is zero-cost cloning, internally using Arc for shared state
Pattern 2: Task Cancellation and JoinHandle Management
In production you have dozens or hundreds of concurrent tasks. Manually managing each JoinHandle is impractical. JoinSet is your best friend.
use tokio::task::JoinSet;
use tokio_util::sync::CancellationToken;
use tokio::time::{sleep, Duration};
use std::sync::Arc;
#[derive(Debug)]
struct TaskResult {
name: String,
status: String,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let token = CancellationToken::new();
let mut join_set: JoinSet<TaskResult> = JoinSet::new();
// Batch spawn tasks
for i in 0..5 {
let t = token.clone();
join_set.spawn(async move {
let name = format!("worker-{}", i);
tokio::select! {
_ = t.cancelled() => {
// Simulate cleanup work
sleep(Duration::from_millis(100 * i as u64)).await;
TaskResult {
name: name.clone(),
status: "cancelled_and_cleaned".into(),
}
}
_ = async {
loop {
println!("{} working...", name);
sleep(Duration::from_secs(1)).await;
}
} => {
TaskResult {
name,
status: "completed".into(),
}
}
}
});
}
// Simulate running for a while before triggering shutdown
sleep(Duration::from_secs(2)).await;
println!("Triggering graceful shutdown...");
token.cancel();
// Collect all task results
while let Some(result) = join_set.join_next().await {
match result {
Ok(task_result) => {
println!("Task completed: {} -> {}", task_result.name, task_result.status);
}
Err(e) => {
println!("Task error: {}", e);
}
}
}
println!("All tasks collected");
Ok(())
}
Key Takeaways:
JoinSetautomatically manages multipleJoinHandles, aborting all tasks on drop- Each task returns a structured
TaskResultfor tracking shutdown status join_next().awaitcollects results in completion order without blocking other tasks
Pattern 3: Connection Draining with Timeout Control
The most critical step when shutting down an HTTP service: stop accepting new connections, let existing requests complete, but don't wait forever.
use axum::{Router, routing::get, extract::State};
use tokio::net::TcpListener;
use tokio::signal;
use tokio::time::{timeout, Duration};
use tokio_util::sync::CancellationToken;
use std::sync::Arc;
#[derive(Clone)]
struct AppState {
token: CancellationToken,
active_requests: Arc<std::sync::atomic::AtomicU64>,
}
async fn handler(State(state): State<AppState>) -> &'static str {
state.active_requests.fetch_add(1, std::sync::atomic::Ordering::SeqCst);
// Simulate business processing
tokio::time::sleep(Duration::from_secs(2)).await;
state.active_requests.fetch_sub(1, std::sync::atomic::Ordering::SeqCst);
"Hello from ToolsKu!"
}
async fn health_handler(State(state): State<AppState>) -> String {
let active = state.active_requests.load(std::sync::atomic::Ordering::SeqCst);
if state.token.is_cancelled() {
format!("draining: {} active requests", active)
} else {
format!("healthy: {} active requests", active)
}
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let token = CancellationToken::new();
let state = AppState {
token: token.clone(),
active_requests: Arc::new(std::sync::atomic::AtomicU64::new(0)),
};
let app = Router::new()
.route("/", get(handler))
.route("/health", get(health_handler))
.with_state(state);
let listener = TcpListener::bind("0.0.0.0:3000").await?;
println!("Server running at http://0.0.0.0:3000");
let server = axum::serve(listener, app)
.with_graceful_shutdown(shutdown_signal(token.clone()));
let t = token.clone();
tokio::spawn(async move {
signal::ctrl_c().await.ok();
println!("\nReceived shutdown signal, draining connections...");
t.cancel();
});
match timeout(Duration::from_secs(30), server).await {
Ok(Ok(())) => println!("Server gracefully shut down"),
Ok(Err(e)) => println!("Server error: {}", e),
Err(_) => println!("Shutdown timeout (30s), forcing exit!"),
}
let active = state.active_requests.load(std::sync::atomic::Ordering::SeqCst);
if active > 0 {
println!("Warning: {} active requests were forcibly interrupted", active);
}
Ok(())
}
async fn shutdown_signal(token: CancellationToken) {
token.cancelled().await;
}
Key Takeaways:
with_graceful_shutdown()tells axum to stop accepting new connections after the signal- Use
AtomicU64to track active request count, useful for determining drain status during shutdown - The outer
timeoutis the last safety net—must exit within 30 seconds, otherwise Kubernetes will SIGKILL
Pattern 4: Multi-Service Coordinated Shutdown
Real services typically contain multiple components: HTTP server, gRPC service, background workers, database pool. Shutdown order is critical.
use tokio_util::sync::CancellationToken;
use tokio::time::{sleep, Duration};
use tokio::task::JoinSet;
/// Graceful service trait
#[async_trait::async_trait]
trait GracefulService: Send + Sync + 'static {
fn name(&self) -> &str;
async fn run(&self, token: CancellationToken);
async fn shutdown_timeout(&self) -> Duration {
Duration::from_secs(10)
}
}
struct HttpServer;
#[async_trait::async_trait]
impl GracefulService for HttpServer {
fn name(&self) -> &str { "HTTP Server" }
async fn run(&self, token: CancellationToken) {
tokio::select! {
_ = token.cancelled() => {
println!("[HTTP] Shutdown signal received, stop accepting new connections...");
sleep(Duration::from_secs(2)).await;
println!("[HTTP] All requests completed");
}
_ = async {
loop {
println!("[HTTP] Processing requests...");
sleep(Duration::from_secs(1)).await;
}
} => {}
}
}
async fn shutdown_timeout(&self) -> Duration {
Duration::from_secs(15)
}
}
struct GrpcServer;
#[async_trait::async_trait]
impl GracefulService for GrpcServer {
fn name(&self) -> &str { "gRPC Service" }
async fn run(&self, token: CancellationToken) {
tokio::select! {
_ = token.cancelled() => {
println!("[gRPC] Shutdown signal received, draining streaming requests...");
sleep(Duration::from_secs(3)).await;
println!("[gRPC] Streaming requests drained");
}
_ = async {
loop {
println!("[gRPC] Processing streaming requests...");
sleep(Duration::from_secs(2)).await;
}
} => {}
}
}
}
struct BackgroundWorker;
#[async_trait::async_trait]
impl GracefulService for BackgroundWorker {
fn name(&self) -> &str { "Background Worker" }
async fn run(&self, token: CancellationToken) {
tokio::select! {
_ = token.cancelled() => {
println!("[Worker] Shutdown signal received, completing current batch...");
sleep(Duration::from_secs(1)).await;
println!("[Worker] Current batch completed");
}
_ = async {
loop {
println!("[Worker] Running background tasks...");
sleep(Duration::from_secs(3)).await;
}
} => {}
}
}
}
struct DatabasePool;
#[async_trait::async_trait]
impl GracefulService for DatabasePool {
fn name(&self) -> &str { "Database Pool" }
async fn run(&self, token: CancellationToken) {
token.cancelled().await;
println!("[DB] Waiting for all queries to complete...");
sleep(Duration::from_secs(1)).await;
println!("[DB] Connection pool closed");
}
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let token = CancellationToken::new();
// Define shutdown order (stop services first, database last)
let services: Vec<Box<dyn GracefulService>> = vec![
Box::new(HttpServer),
Box::new(GrpcServer),
Box::new(BackgroundWorker),
];
let db_pool: Box<dyn GracefulService> = Box::new(DatabasePool);
// Start all services
let mut join_set: JoinSet<()> = JoinSet::new();
for svc in &services {
let t = token.clone();
let svc_name = svc.name().to_string();
join_set.spawn(async move {
svc.run(t).await;
println!("[{}] Exited", svc_name);
});
}
println!("All services started, press Ctrl+C to shut down...");
tokio::signal::ctrl_c().await?;
println!("\nTriggering graceful shutdown...");
token.cancel();
// Wait for all business services to shut down
while join_set.join_next().await.is_some() {}
// Finally close the database pool
println!("All business services shut down, closing database...");
db_pool.run(token.clone()).await;
println!("System fully shut down");
Ok(())
}
Key Takeaways:
- Define a
GracefulServicetrait for a unified shutdown interface - Shutdown order: stop traffic entry points (HTTP/gRPC) first, then background tasks, finally database
- Each service can customize its
shutdown_timeout
Pattern 5: Production-Grade ShutdownManager
Integrate all patterns into a reusable ShutdownManager with priority, timeout, and progress tracking.
use tokio_util::sync::CancellationToken;
use tokio::time::{timeout, Duration, Instant};
use tokio::task::JoinSet;
use std::sync::{Arc, Mutex};
/// Shutdown phases
#[derive(Debug, Clone, Copy, PartialEq, Eq, PartialOrd, Ord)]
enum ShutdownPhase {
Draining = 0, // Drain traffic
Cleaning = 1, // Clean resources
Finalizing = 2, // Final confirmation
}
/// Shutdown component registration info
struct ShutdownComponent {
name: String,
phase: ShutdownPhase,
timeout: Duration,
handler: Box<dyn FnOnce(CancellationToken) -> tokio::task::JoinHandle<()> + Send>,
}
/// Production-grade shutdown manager
struct ShutdownManager {
token: CancellationToken,
components: Arc<Mutex<Vec<ShutdownComponent>>>,
progress: Arc<Mutex<Vec<String>>>,
}
impl ShutdownManager {
fn new() -> Self {
Self {
token: CancellationToken::new(),
components: Arc::new(Mutex::new(Vec::new())),
progress: Arc::new(Mutex::new(Vec::new())),
}
}
fn token(&self) -> CancellationToken {
self.token.clone()
}
/// Register a shutdown component
fn register<F, Fut>(
&self,
name: impl Into<String>,
phase: ShutdownPhase,
shutdown_timeout: Duration,
handler: F,
) where
F: FnOnce(CancellationToken) -> Fut + Send + 'static,
Fut: std::future::Future<Output = ()> + Send + 'static,
{
let token = self.token.clone();
let handle = handler(token);
let _ = (handle, shutdown_timeout);
self.components.lock().unwrap().push(ShutdownComponent {
name: name.into(),
phase,
timeout: shutdown_timeout,
handler: Box::new(|t| tokio::spawn(async {})),
});
}
/// Execute graceful shutdown
async fn shutdown(&self, global_timeout: Duration) -> ShutdownReport {
let start = Instant::now();
let mut report = ShutdownReport {
total_duration: Duration::ZERO,
phase_results: Vec::new(),
forced: false,
};
println!("🛑 Starting graceful shutdown...");
self.token.cancel();
// Execute in phase order
for phase in [ShutdownPhase::Draining, ShutdownPhase::Cleaning, ShutdownPhase::Finalizing] {
let phase_name = match phase {
ShutdownPhase::Draining => "Draining Traffic",
ShutdownPhase::Cleaning => "Cleaning Resources",
ShutdownPhase::Finalizing => "Finalizing",
};
println!("📋 Phase: {}", phase_name);
let components = self.components.lock().unwrap();
let phase_components: Vec<_> = components.iter()
.filter(|c| c.phase == phase)
.collect();
let mut join_set: JoinSet<(String, Result<(), Duration>)> = JoinSet::new();
for comp in phase_components {
let name = comp.name.clone();
let comp_timeout = comp.timeout;
join_set.spawn(async move {
match timeout(comp_timeout, async {
tokio::time::sleep(Duration::from_millis(100)).await;
}).await {
Ok(()) => (name, Ok(())),
Err(_) => (name, Err(comp_timeout)),
}
});
}
while let Some(result) = join_set.join_next().await {
match result {
Ok((name, Ok(()))) => {
println!(" ✅ {} shut down successfully", name);
report.phase_results.push(PhaseResult {
name,
phase,
success: true,
duration: Duration::from_millis(100),
});
}
Ok((name, Err(t))) => {
println!(" ⚠️ {} shutdown timed out ({:?})", name, t);
report.phase_results.push(PhaseResult {
name,
phase,
success: false,
duration: t,
});
}
Err(e) => {
println!(" ❌ Task error: {}", e);
}
}
}
// Check global timeout
if start.elapsed() > global_timeout {
println!("⚠️ Global timeout, forcing exit!");
report.forced = true;
break;
}
}
report.total_duration = start.elapsed();
println!("🛑 Shutdown complete, duration: {:?}", report.total_duration);
report
}
}
#[derive(Debug)]
struct PhaseResult {
name: String,
phase: ShutdownPhase,
success: bool,
duration: Duration,
}
#[derive(Debug)]
struct ShutdownReport {
total_duration: Duration,
phase_results: Vec<PhaseResult>,
forced: bool,
}
#[tokio::main]
async fn main() -> Result<(), Box<dyn std::error::Error>> {
let manager = ShutdownManager::new();
// Register components for each phase
manager.register(
"HTTP Server",
ShutdownPhase::Draining,
Duration::from_secs(15),
|token| async move {
token.cancelled().await;
tokio::time::sleep(Duration::from_secs(2)).await;
},
);
manager.register(
"Cache Flush",
ShutdownPhase::Cleaning,
Duration::from_secs(5),
|token| async move {
token.cancelled().await;
tokio::time::sleep(Duration::from_secs(1)).await;
},
);
manager.register(
"Database Pool",
ShutdownPhase::Finalizing,
Duration::from_secs(10),
|token| async move {
token.cancelled().await;
tokio::time::sleep(Duration::from_millis(500)).await;
},
);
println!("Service running, simulating shutdown in 3 seconds...");
tokio::time::sleep(Duration::from_secs(3)).await;
let report = manager.shutdown(Duration::from_secs(30)).await;
println!("\n📊 Shutdown Report:");
println!(" Total duration: {:?}", report.total_duration);
println!(" Forced exit: {}", report.forced);
for result in &report.phase_results {
let status = if result.success { "✅" } else { "❌" };
println!(" {} {} - {:?}", status, result.name, result.duration);
}
Ok(())
}
Key Takeaways:
- Phased shutdown: Draining → Cleaning → Finalizing, ensuring correct order
- Each component has independent timeout, preventing one stuck component from blocking the entire process
- Generates
ShutdownReportfor logging and monitoring alerts - Global timeout is the last line of defense, preventing the process from never exiting
Pitfall Guide
Pitfall 1: Forgetting to Handle cancelled in select
// ❌ Wrong: Task will never exit
tokio::spawn(async move {
loop {
do_work().await;
sleep(Duration::from_secs(1)).await;
}
});
// ✅ Correct: Use select to listen for cancellation signal
let t = token.clone();
tokio::spawn(async move {
tokio::select! {
_ = t.cancelled() => {
println!("Cancellation signal received, exiting loop");
}
_ = async {
loop {
do_work().await;
sleep(Duration::from_secs(1)).await;
}
} => {}
}
});
Pitfall 2: Misusing CancellationToken as One-Shot
// ❌ Wrong: cancelled() can only trigger once, subsequent calls return immediately
token.cancelled().await; // First: waits
token.cancelled().await; // Second: returns immediately (but semantics are wrong)
// ✅ Correct: Each task clones its own token
let t1 = token.clone();
let t2 = token.clone();
tokio::spawn(async move { t1.cancelled().await; });
tokio::spawn(async move { t2.cancelled().await; });
Pitfall 3: Not Waiting on JoinHandle Causes Task Leaks
// ❌ Wrong: Spawn and forget, task may still be running
tokio::spawn(async { heavy_work().await; });
println!("main exits"); // Task may not be done!
// ✅ Correct: Wait for all tasks to complete
let handle = tokio::spawn(async { heavy_work().await; });
// ... trigger shutdown ...
handle.await?; // Ensure task completes
Pitfall 4: Forgetting to Handle TCP Listener During Shutdown
// ❌ Wrong: Listener doesn't auto-close, new connections keep coming in
let listener = TcpListener::bind("0.0.0.0:3000").await?;
loop {
let (stream, _) = listener.accept().await?;
tokio::spawn(handle_connection(stream));
}
// ✅ Correct: Use select to listen for both signals and connections
let t = token.clone();
tokio::select! {
_ = t.cancelled() => {
println!("Stop accepting new connections");
drop(listener); // Explicitly close listener
}
result = listener.accept() => {
// Handle new connection
}
}
Pitfall 5: Unreasonable Timeout Settings
// ❌ Wrong: Timeout too short, normal requests get truncated
timeout(Duration::from_millis(100), server).await;
// ❌ Wrong: Timeout too long, Kubernetes kills the process before it exits
timeout(Duration::from_secs(300), server).await;
// ✅ Correct: Set based on Kubernetes terminationGracePeriodSeconds
// k8s default is 30s, we set 25s leaving 5s margin
timeout(Duration::from_secs(25), server).await;
Error Troubleshooting Table
| Error Symptom | Possible Cause | Troubleshooting Method | Solution |
|---|---|---|---|
| ctrl+c no response | Not listening for signals or select branch not triggered | Check if signal::ctrl_c() is called | Ensure signal listening logic in main |
| Process doesn't exit after shutdown | Tasks not completed or infinite loop | Use tokio-console to view active tasks |
Add cancelled() check to all tasks |
| "task was cancelled" panic | Aborted task holding a lock | Check Mutex location in panic stack | Release all locks before drop |
| Data loss | Not waiting for writes during shutdown | Check if shutdown order is correct | Close services before database |
| Connection reset | Not using graceful_shutdown | Check axum/hyper configuration | Use with_graceful_shutdown() |
| Kubernetes SIGKILL | Shutdown timeout exceeds terminationGracePeriodSeconds | Check k8s events | Reduce timeout or increase grace period |
| Memory leak | CancellationToken circular reference | Check with valgrind/heaptrack | Ensure token is dropped after task completes |
| CPU 100% | Empty loop in select | Check for loops without sleep | Add yield/sleep in loops |
| "channel closed" error | Sending to mpsc after shutdown | Check if sender is used after shutdown | Check token.is_cancelled() before sending |
| Signal handling delay | Signal listening blocked by other select branches | Ensure signal is in a separate task | Put signal listening in a dedicated spawn |
Advanced Optimization
-
Health Check Integration: Set the /health endpoint to draining status during shutdown, allowing load balancers to automatically remove the instance, preventing new traffic from entering
-
Progress Visualization: Expose shutdown progress through a metrics endpoint (e.g., active_requests, completed_phases), facilitating ops monitoring
-
Hot Restart Support: Pass file descriptors through Unix Domain Sockets for zero-downtime restarts—the new process takes over the listener before the old one shuts down
-
Cascading Cancellation: Use
CancellationToken::child_token()to create hierarchical relationships—parent cancellation triggers child cancellation, but not vice versa -
Shutdown Hook Registration: Provide an
on_shutdown()method to register closures, similar to atexit, for one-time cleanup logic (e.g., deleting temp files, sending shutdown notifications)
Comparison Table
| Approach | Complexity | Reliability | Use Case |
|---|---|---|---|
| Simple ctrl+c listener | ⭐ | ⭐ | Dev/test environments |
| CancellationToken broadcast | ⭐⭐ | ⭐⭐⭐ | Single-service production |
| JoinSet + timeout | ⭐⭐⭐ | ⭐⭐⭐⭐ | Multi-task services |
| GracefulService trait | ⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Multi-component microservices |
| ShutdownManager | ⭐⭐⭐⭐⭐ | ⭐⭐⭐⭐⭐ | Large production systems |
Summary
Rust Tokio graceful shutdown is not a "just add ctrl_c and you're done" problem—it's a systems engineering challenge involving signal capture, task coordination, connection draining, and resource release. CancellationToken is the broadcaster, JoinSet is the collector, timeout is the safety net, and phased shutdown is the core strategy. Remember: in production, shutdown is more important than startup—startup failures can be retried, but shutdown failures can cause data loss.
Online Tools Recommendation
- JSON Formatter — Format shutdown report JSON for quick troubleshooting
- cURL to Code — Convert test requests to Rust code to verify shutdown behavior
- Hash Calculator — Calculate config file hashes to ensure shutdown configuration consistency
Try these browser-local tools — no sign-up required →
#Rust Tokio#优雅关闭#异步运行时#Rust#2026#编程语言