Keyboard shortcuts

Press ← or β†’ to navigate between chapters

Press S or / to search in the book

Press ? to show this help

Press Esc to hide this help

Retry Mechanism

Status: βœ… Implemented (v0.2.0) Module: yanshi-core::retry

Overview

Yanshi provides a comprehensive retry mechanism for handling transient failures in workflow execution. The retry system supports multiple strategies, selective error matching, and detailed error context tracking.

Features

  • πŸ”„ Multiple retry strategies: Fixed delay, exponential backoff, linear backoff
  • 🎯 Selective retries: Only retry specific error types
  • ⏱️ Configurable limits: Max attempts and total duration
  • πŸ“Š Detailed error context: Full error chains and execution history
  • 🎲 Jitter support: Prevent thundering herd problem
  • πŸ” Observability: Optional tracing integration

Quick Start

#![allow(unused)]
fn main() {
use yanshi_core::{RetryPolicy, RetryStrategy, execute_with_retry};

let policy = RetryPolicy::builder()
    .max_attempts(3)
    .strategy(RetryStrategy::exponential_backoff(100, 5000, 2.0))
    .build();

let result = execute_with_retry(&policy, "my_operation", || async {
    // Your async operation here
    Ok("Success")
}).await?;
}

Retry Strategies

1. Fixed Delay

Constant delay between retry attempts.

#![allow(unused)]
fn main() {
use yanshi_core::RetryStrategy;

let strategy = RetryStrategy::fixed(1000); // 1 second delay
}

Use case: Simple scenarios where backoff is not needed

2. Exponential Backoff

Delay increases exponentially with optional jitter.

#![allow(unused)]
fn main() {
let strategy = RetryStrategy::exponential_backoff(
    100,   // initial_delay_ms: 100ms
    10000, // max_delay_ms: 10 seconds
    2.0,   // multiplier: 2x each attempt
);
// Delays: 100ms β†’ 200ms β†’ 400ms β†’ 800ms β†’ ... β†’ 10s (max)
}

Use case: API rate limiting, network failures, external service calls

Jitter: Automatically enabled to prevent synchronized retries

3. Linear Backoff

Delay increases linearly.

#![allow(unused)]
fn main() {
let strategy = RetryStrategy::linear(
    100, // initial_delay_ms: 100ms
    50,  // increment_ms: +50ms per attempt
);
// Delays: 100ms β†’ 150ms β†’ 200ms β†’ 250ms β†’ ...
}

Use case: Gradual backoff when exponential is too aggressive

Retry Policies

Basic Configuration

#![allow(unused)]
fn main() {
use yanshi_core::{RetryPolicy, RetryStrategy};
use std::time::Duration;

let policy = RetryPolicy::builder()
    .max_attempts(5)
    .strategy(RetryStrategy::exponential_backoff(100, 10000, 2.0))
    .max_duration(Duration::from_secs(300)) // Max 5 minutes total
    .build();
}

Selective Retries

Only retry specific error types:

#![allow(unused)]
fn main() {
use yanshi_core::ErrorPattern;

let policy = RetryPolicy::builder()
    .max_attempts(3)
    .strategy(RetryStrategy::fixed(1000))
    // Only retry these error types
    .retryable_error(ErrorPattern::NetworkError)
    .retryable_error(ErrorPattern::TimeoutError)
    .retryable_error(ErrorPattern::RateLimitError)
    .build();
}

Error Patterns

Available error patterns:

#![allow(unused)]
fn main() {
// Predefined patterns
ErrorPattern::NetworkError          // Network connection issues
ErrorPattern::TimeoutError          // Operation timeouts
ErrorPattern::RateLimitError        // Rate limit (429) errors
ErrorPattern::ServiceUnavailable    // Service unavailable (503) errors

// Custom patterns
ErrorPattern::ErrorType {
    name: "ConfigurationError".to_string()
}
ErrorPattern::MessageContains {
    text: "temporary failure".to_string()
}
}

Error Context

Get detailed error information with context:

#![allow(unused)]
fn main() {
use yanshi_core::execute_with_retry_and_context;

let result = execute_with_retry_and_context(
    &policy,
    "run-id-123",      // Workflow run ID
    "api_call_node",   // Node name
    Some("http"),      // Node type (optional)
    || async {
        // Your operation
        perform_api_call().await
    }
).await;

match result {
    Ok(value) => println!("Success: {:?}", value),
    Err((error, context)) => {
        println!("Error Summary: {}", context.summary());
        println!("Detailed Report:\n{}", context.detailed_report());
    }
}
}

Error Context Fields

  • run_id: Workflow run identifier
  • node_name: Name of the failed node
  • node_type: Type of node (e.g., β€œhttp”, β€œllm”)
  • timestamp: When the error occurred
  • error_chain: Complete chain of errors
  • duration: How long the operation ran
  • retry_attempt: Which retry attempt failed
  • execution_history: List of successful nodes before failure
  • metadata: Additional debugging information

YAML Configuration

Configure retry policies in workflow YAML files:

nodes:
  - name: api_call
    type: http
    url: "https://api.example.com/data"
    retry:
      max_attempts: 3
      strategy:
        type: exponential_backoff
        initial_delay_ms: 100
        max_delay_ms: 5000
        multiplier: 2.0
        jitter: true
      retryable_errors:
        - NetworkError
        - TimeoutError
        - MessageContains: "503"
      max_duration: 30s

Best Practices

1. Choose Appropriate Strategy

  • Fixed delay: Simple, predictable scenarios
  • Exponential backoff: External APIs, rate limiting
  • Linear backoff: Moderate growth when exponential is too aggressive

2. Set Reasonable Limits

#![allow(unused)]
fn main() {
let policy = RetryPolicy::builder()
    .max_attempts(5)                          // Don't retry indefinitely
    .max_duration(Duration::from_secs(60))    // Prevent hanging
    .strategy(RetryStrategy::exponential_backoff(100, 10000, 2.0))
    .build();
}

3. Be Selective About Retries

#![allow(unused)]
fn main() {
// βœ… Good: Only retry transient errors
.retryable_error(ErrorPattern::NetworkError)
.retryable_error(ErrorPattern::TimeoutError)

// ❌ Bad: Retrying configuration errors won't help
// ConfigurationError should fail fast
}

4. Use Error Context for Debugging

Always capture error context in production:

#![allow(unused)]
fn main() {
let result = execute_with_retry_and_context(
    &policy, run_id, node_name, node_type, operation
).await;

if let Err((_, context)) = result {
    // Log detailed report for debugging
    eprintln!("{}", context.detailed_report());
}
}

5. Monitor Retry Metrics

Retry attempts, exhaustion, and success-after-retry are logged automatically via tracing (unconditional as of W0.7 β€” no feature flag needed). Wire up a tracing subscriber in your binary to see them.

Examples

See examples/retry_example.rs for comprehensive usage examples:

cargo run --example retry_example

Performance Considerations

Memory

  • Retry context is lightweight (~few KB)
  • Error chains are cloned, not deep-copied
  • Input sanitization limits large value storage

CPU

  • Retry overhead: < 5ms per attempt
  • Jitter calculation: < 1ΞΌs
  • Strategy delay calculation: constant time

Network

  • Exponential backoff prevents overwhelming services
  • Jitter prevents synchronized retry storms
  • Max duration prevents indefinite retries

Limitations

  1. Not distributed: Retry state is per-process
  2. No persistence: Retry context lost on crash
  3. Synchronous retry: Doesn’t queue for later
  4. No circuit breaking: Use robustness module for that

Integration Examples

With Workflow Nodes

#![allow(unused)]
fn main() {
use yanshi_core::{AsyncNode, AsyncNodeInputs, AsyncNodeResult};

struct HttpNode {
    url: String,
    retry_policy: RetryPolicy,
}

#[async_trait]
impl AsyncNode for HttpNode {
    async fn execute(&self, inputs: &AsyncNodeInputs) -> AsyncNodeResult {
        execute_with_retry(&self.retry_policy, "http_request", || async {
            // Perform HTTP request
            make_http_request(&self.url).await
        }).await
    }
}
}

With LLM Calls

#![allow(unused)]
fn main() {
let policy = RetryPolicy::builder()
    .max_attempts(3)
    .strategy(RetryStrategy::exponential_backoff(1000, 30000, 2.0))
    .retryable_error(ErrorPattern::RateLimitError)
    .retryable_error(ErrorPattern::ServiceUnavailable)
    .build();

let response = execute_with_retry(&policy, "llm_call", || async {
    llm_client.complete("Your prompt").await
}).await?;
}

Testing

Test retry behavior in your workflows:

#![allow(unused)]
fn main() {
#[tokio::test]
async fn test_retry_on_transient_failure() {
    let attempt_counter = Arc::new(AtomicU32::new(0));

    let policy = RetryPolicy::builder()
        .max_attempts(3)
        .strategy(RetryStrategy::fixed(10))
        .build();

    let result = execute_with_retry(&policy, "test", || {
        let counter = attempt_counter.clone();
        async move {
            let attempt = counter.fetch_add(1, Ordering::SeqCst);
            if attempt < 2 {
                Err(YanshiError::NodeExecutionFailed {
                    message: "Transient failure".into()
                })
            } else {
                Ok("Success")
            }
        }
    }).await;

    assert!(result.is_ok());
    assert_eq!(attempt_counter.load(Ordering::SeqCst), 3);
}
}

Troubleshooting

Retries Not Happening

  1. Check error pattern matches:

    #![allow(unused)]
    fn main() {
    policy.is_retryable(&error) // Should return true
    }
  2. Verify max attempts not reached

  3. Check max duration not exceeded

Too Many Retries

  • Reduce max_attempts
  • Add max_duration limit
  • Make error patterns more specific

Slow Retries

  • Use exponential backoff instead of fixed
  • Reduce max_delay_ms
  • Consider circuit breaker pattern

API Reference

See API docs for complete reference:

Changelog

v0.2.0 (2025-10-26)

  • ✨ Initial retry mechanism implementation
  • ✨ Three retry strategies (fixed, exponential, linear)
  • ✨ Selective error matching
  • ✨ Error context tracking
  • ✨ Observability integration
  • πŸ“š Comprehensive documentation and examples

Related: See WORKFLOW_DEBUGGING.md for adjacent workflow reliability features. (The former resource-management guide was archived in W5.3 β€” see archive/RESOURCE_MANAGEMENT_2026-08-12-pre-w5.3.md.)