Retry Mechanism
Status: β
Implemented (v0.2.0)
Module: yanshi-core::retry
Overview
Yanshi provides a comprehensive retry mechanism for handling transient failures in workflow execution. The retry system supports multiple strategies, selective error matching, and detailed error context tracking.
Features
- π Multiple retry strategies: Fixed delay, exponential backoff, linear backoff
- π― Selective retries: Only retry specific error types
- β±οΈ Configurable limits: Max attempts and total duration
- π Detailed error context: Full error chains and execution history
- π² Jitter support: Prevent thundering herd problem
- π Observability: Optional tracing integration
Quick Start
#![allow(unused)]
fn main() {
use yanshi_core::{RetryPolicy, RetryStrategy, execute_with_retry};
let policy = RetryPolicy::builder()
.max_attempts(3)
.strategy(RetryStrategy::exponential_backoff(100, 5000, 2.0))
.build();
let result = execute_with_retry(&policy, "my_operation", || async {
// Your async operation here
Ok("Success")
}).await?;
}
Retry Strategies
1. Fixed Delay
Constant delay between retry attempts.
#![allow(unused)]
fn main() {
use yanshi_core::RetryStrategy;
let strategy = RetryStrategy::fixed(1000); // 1 second delay
}
Use case: Simple scenarios where backoff is not needed
2. Exponential Backoff
Delay increases exponentially with optional jitter.
#![allow(unused)]
fn main() {
let strategy = RetryStrategy::exponential_backoff(
100, // initial_delay_ms: 100ms
10000, // max_delay_ms: 10 seconds
2.0, // multiplier: 2x each attempt
);
// Delays: 100ms β 200ms β 400ms β 800ms β ... β 10s (max)
}
Use case: API rate limiting, network failures, external service calls
Jitter: Automatically enabled to prevent synchronized retries
3. Linear Backoff
Delay increases linearly.
#![allow(unused)]
fn main() {
let strategy = RetryStrategy::linear(
100, // initial_delay_ms: 100ms
50, // increment_ms: +50ms per attempt
);
// Delays: 100ms β 150ms β 200ms β 250ms β ...
}
Use case: Gradual backoff when exponential is too aggressive
Retry Policies
Basic Configuration
#![allow(unused)]
fn main() {
use yanshi_core::{RetryPolicy, RetryStrategy};
use std::time::Duration;
let policy = RetryPolicy::builder()
.max_attempts(5)
.strategy(RetryStrategy::exponential_backoff(100, 10000, 2.0))
.max_duration(Duration::from_secs(300)) // Max 5 minutes total
.build();
}
Selective Retries
Only retry specific error types:
#![allow(unused)]
fn main() {
use yanshi_core::ErrorPattern;
let policy = RetryPolicy::builder()
.max_attempts(3)
.strategy(RetryStrategy::fixed(1000))
// Only retry these error types
.retryable_error(ErrorPattern::NetworkError)
.retryable_error(ErrorPattern::TimeoutError)
.retryable_error(ErrorPattern::RateLimitError)
.build();
}
Error Patterns
Available error patterns:
#![allow(unused)]
fn main() {
// Predefined patterns
ErrorPattern::NetworkError // Network connection issues
ErrorPattern::TimeoutError // Operation timeouts
ErrorPattern::RateLimitError // Rate limit (429) errors
ErrorPattern::ServiceUnavailable // Service unavailable (503) errors
// Custom patterns
ErrorPattern::ErrorType {
name: "ConfigurationError".to_string()
}
ErrorPattern::MessageContains {
text: "temporary failure".to_string()
}
}
Error Context
Get detailed error information with context:
#![allow(unused)]
fn main() {
use yanshi_core::execute_with_retry_and_context;
let result = execute_with_retry_and_context(
&policy,
"run-id-123", // Workflow run ID
"api_call_node", // Node name
Some("http"), // Node type (optional)
|| async {
// Your operation
perform_api_call().await
}
).await;
match result {
Ok(value) => println!("Success: {:?}", value),
Err((error, context)) => {
println!("Error Summary: {}", context.summary());
println!("Detailed Report:\n{}", context.detailed_report());
}
}
}
Error Context Fields
run_id: Workflow run identifiernode_name: Name of the failed nodenode_type: Type of node (e.g., βhttpβ, βllmβ)timestamp: When the error occurrederror_chain: Complete chain of errorsduration: How long the operation ranretry_attempt: Which retry attempt failedexecution_history: List of successful nodes before failuremetadata: Additional debugging information
YAML Configuration
Configure retry policies in workflow YAML files:
nodes:
- name: api_call
type: http
url: "https://api.example.com/data"
retry:
max_attempts: 3
strategy:
type: exponential_backoff
initial_delay_ms: 100
max_delay_ms: 5000
multiplier: 2.0
jitter: true
retryable_errors:
- NetworkError
- TimeoutError
- MessageContains: "503"
max_duration: 30s
Best Practices
1. Choose Appropriate Strategy
- Fixed delay: Simple, predictable scenarios
- Exponential backoff: External APIs, rate limiting
- Linear backoff: Moderate growth when exponential is too aggressive
2. Set Reasonable Limits
#![allow(unused)]
fn main() {
let policy = RetryPolicy::builder()
.max_attempts(5) // Don't retry indefinitely
.max_duration(Duration::from_secs(60)) // Prevent hanging
.strategy(RetryStrategy::exponential_backoff(100, 10000, 2.0))
.build();
}
3. Be Selective About Retries
#![allow(unused)]
fn main() {
// β
Good: Only retry transient errors
.retryable_error(ErrorPattern::NetworkError)
.retryable_error(ErrorPattern::TimeoutError)
// β Bad: Retrying configuration errors won't help
// ConfigurationError should fail fast
}
4. Use Error Context for Debugging
Always capture error context in production:
#![allow(unused)]
fn main() {
let result = execute_with_retry_and_context(
&policy, run_id, node_name, node_type, operation
).await;
if let Err((_, context)) = result {
// Log detailed report for debugging
eprintln!("{}", context.detailed_report());
}
}
5. Monitor Retry Metrics
Retry attempts, exhaustion, and success-after-retry are logged
automatically via tracing (unconditional as of W0.7 β no feature flag
needed). Wire up a tracing subscriber in your binary to see them.
Examples
See examples/retry_example.rs for comprehensive usage examples:
cargo run --example retry_example
Performance Considerations
Memory
- Retry context is lightweight (~few KB)
- Error chains are cloned, not deep-copied
- Input sanitization limits large value storage
CPU
- Retry overhead: < 5ms per attempt
- Jitter calculation: < 1ΞΌs
- Strategy delay calculation: constant time
Network
- Exponential backoff prevents overwhelming services
- Jitter prevents synchronized retry storms
- Max duration prevents indefinite retries
Limitations
- Not distributed: Retry state is per-process
- No persistence: Retry context lost on crash
- Synchronous retry: Doesnβt queue for later
- No circuit breaking: Use
robustnessmodule for that
Integration Examples
With Workflow Nodes
#![allow(unused)]
fn main() {
use yanshi_core::{AsyncNode, AsyncNodeInputs, AsyncNodeResult};
struct HttpNode {
url: String,
retry_policy: RetryPolicy,
}
#[async_trait]
impl AsyncNode for HttpNode {
async fn execute(&self, inputs: &AsyncNodeInputs) -> AsyncNodeResult {
execute_with_retry(&self.retry_policy, "http_request", || async {
// Perform HTTP request
make_http_request(&self.url).await
}).await
}
}
}
With LLM Calls
#![allow(unused)]
fn main() {
let policy = RetryPolicy::builder()
.max_attempts(3)
.strategy(RetryStrategy::exponential_backoff(1000, 30000, 2.0))
.retryable_error(ErrorPattern::RateLimitError)
.retryable_error(ErrorPattern::ServiceUnavailable)
.build();
let response = execute_with_retry(&policy, "llm_call", || async {
llm_client.complete("Your prompt").await
}).await?;
}
Testing
Test retry behavior in your workflows:
#![allow(unused)]
fn main() {
#[tokio::test]
async fn test_retry_on_transient_failure() {
let attempt_counter = Arc::new(AtomicU32::new(0));
let policy = RetryPolicy::builder()
.max_attempts(3)
.strategy(RetryStrategy::fixed(10))
.build();
let result = execute_with_retry(&policy, "test", || {
let counter = attempt_counter.clone();
async move {
let attempt = counter.fetch_add(1, Ordering::SeqCst);
if attempt < 2 {
Err(YanshiError::NodeExecutionFailed {
message: "Transient failure".into()
})
} else {
Ok("Success")
}
}
}).await;
assert!(result.is_ok());
assert_eq!(attempt_counter.load(Ordering::SeqCst), 3);
}
}
Troubleshooting
Retries Not Happening
-
Check error pattern matches:
#![allow(unused)] fn main() { policy.is_retryable(&error) // Should return true } -
Verify max attempts not reached
-
Check max duration not exceeded
Too Many Retries
- Reduce
max_attempts - Add
max_durationlimit - Make error patterns more specific
Slow Retries
- Use exponential backoff instead of fixed
- Reduce
max_delay_ms - Consider circuit breaker pattern
API Reference
See API docs for complete reference:
Changelog
v0.2.0 (2025-10-26)
- β¨ Initial retry mechanism implementation
- β¨ Three retry strategies (fixed, exponential, linear)
- β¨ Selective error matching
- β¨ Error context tracking
- β¨ Observability integration
- π Comprehensive documentation and examples
Related: See WORKFLOW_DEBUGGING.md for adjacent workflow reliability features. (The former resource-management guide was archived in W5.3 β see archive/RESOURCE_MANAGEMENT_2026-08-12-pre-w5.3.md.)