淘客系统的容灾演练与恢复：Java Chaos Monkey模拟节点故障下的服务降级与快速切换实践
淘客系统的容灾演练与恢复：Java Chaos Monkey模拟节点故障下的服务降级与快速切换实践
大家好，我是 微赚淘客系统3.0 的研发者省赚客！
高可用是淘客系统的核心要求。为验证在数据库宕机、缓存失效或服务实例崩溃等异常场景下系统的自愈能力，我们引入 Chaos Monkey 思想，结合 Spring Boot Actuator 与自研故障注入模块，在预发环境主动制造故障，并通过 Sentinel 与多活架构实现服务降级与快速切换。
Chaos Monkey 故障注入模块设计
我们基于 Spring Boot 开发轻量级 Chaos Agent，支持运行时注入延迟、异常或服务中断：
packagejuwatech.cn.chaos;importorg.springframework.beans.factory.annotation.Value;importorg.springframework.stereotype.Component;importjava.util.Random;@ComponentpublicclassChaosInjector{@Value("${chaos.enabled:false}")privatebooleanchaosEnabled;@Value("${chaos.failure.rate:0.1}")privatedoublefailureRate;privatefinalRandomrandom=newRandom();publicvoidmaybeThrowException(StringserviceName){if(!chaosEnabled)return;if(random.nextDouble()<failureRate){juwatech.cn.util.AsyncLogger.logAsync("Chaos injected: killing "+serviceName);thrownewRuntimeException("Simulated failure in "+serviceName);}}publicvoidmaybeDelay(longmaxDelayMs){if(!chaosEnabled)return;if(random.nextDouble()<failureRate){longdelay=random.nextInt((int)maxDelayMs);try{Thread.sleep(delay);}catch(InterruptedExceptione){Thread.currentThread().interrupt();}}}}在关键服务中嵌入故障点：
packagejuwatech.cn.service;importjuwatech.cn.chaos.ChaosInjector;importorg.springframework.stereotype.Service;@ServicepublicclassCouponRemoteService{privatefinalChaosInjectorchaosInjector;publicCouponRemoteService(ChaosInjectorchaosInjector){this.chaosInjector=chaosInjector;}publicStringfetchCouponFromRemote(StringitemId){// 模拟调用第三方接口前注入故障chaosInjector.maybeThrowException("coupon-remote-api");chaosInjector.maybeDelay(2000);// 实际调用逻辑（略）return"CPN_"+itemId;}}Sentinel 服务降级规则配置
当远程服务不可用时，自动触发 fallback：
packagejuwatech.cn.fallback;importcom.alibaba.csp.sentinel.slots.block.BlockException;publicclassCouponFallback{publicstaticStringremoteFallback(StringitemId,BlockExceptionex){juwatech.cn.util.AsyncLogger.logAsync("Fallback triggered for item: "+itemId);return"DEFAULT_COUPON_5YUAN";// 返回兜底优惠券}publicstaticStringdbFallback(StringuserId,BlockExceptionex){return"[]";// 返回空列表}}通过 Sentinel 注解绑定资源与降级策略：
packagejuwatech.cn.service;importcom.alibaba.csp.sentinel.annotation.SentinelResource;importjuwatech.cn.fallback.CouponFallback;importorg.springframework.stereotype.Service;@ServicepublicclassCouponQueryService{@SentinelResource(value="fetchCouponFromRemote",blockHandler="remoteFallback",blockHandlerClass=CouponFallback.class)publicStringqueryCoupon(StringitemId){returnnewCouponRemoteService(newjuwatech.cn.chaos.ChaosInjector()).fetchCouponFromRemote(itemId);}}动态加载降级规则（启动时初始化）：
packagejuwatech.cn.config;importcom.alibaba.csp.sentinel.slots.block.degrade.DegradeRule;importcom.alibaba.csp.sentinel.slots.block.degrade.DegradeRuleManager;importorg.springframework.context.annotation.Configuration;importjavax.annotation.PostConstruct;importjava.util.Collections;@ConfigurationpublicclassSentinelDegradeConfig{@PostConstructpublicvoidinitDegradeRules(){DegradeRulerule=newDegradeRule("fetchCouponFromRemote").setGrade(com.alibaba.csp.sentinel.slots.block.degrade.DegradeRule.RT)// 基于响应时间.setCount(200)// 超过200ms视为慢调用.setTimeWindow(10)// 熔断10秒.setMinRequestAmount(5)// 最小请求数.setSlowRatioThreshold(0.5);// 慢调用比例阈值DegradeRuleManager.loadRules(Collections.singletonList(rule));}}多活数据库快速切换机制
主库故障时，自动切换至只读副本。我们封装数据源路由逻辑：
packagejuwatech.cn.datasource;importorg.springframework.jdbc.datasource.lookup.AbstractRoutingDataSource;publicclassRoutingDataSourceextendsAbstractRoutingDataSource{@OverrideprotectedObjectdetermineCurrentLookupKey(){returnDataSourceContext.getDataSourceType();}}上下文管理：
packagejuwatech.cn.datasource;publicclassDataSourceContext{privatestaticfinalThreadLocal<String>contextHolder=newThreadLocal<>();publicstaticvoidsetMaster(){contextHolder.set("master");}publicstaticvoidsetSlave(){contextHolder.set("slave");}publicstaticStringgetDataSourceType(){returncontextHolder.get()!=null?contextHolder.get():"master";}}在数据库操作前检测主库健康状态：
packagejuwatech.cn.service;importjuwatech.cn.datasource.DataSourceContext;importorg.springframework.stereotype.Service;@ServicepublicclassOrderService{privatefinaljuwatech.cn.db.HealthCheckerhealthChecker;publicOrderService(juwatech.cn.db.HealthCheckerhealthChecker){this.healthChecker=healthChecker;}publicvoidcreateOrder(StringorderId){if(!healthChecker.isMasterDbHealthy()){DataSourceContext.setSlave();// 临时切到从库（仅限查询）thrownewIllegalStateException("Master DB down, write operation blocked");}DataSourceContext.setMaster();// 执行写入}}健康检查实现：
packagejuwatech.cn.db;importorg.springframework.jdbc.core.JdbcTemplate;importorg.springframework.stereotype.Component;@ComponentpublicclassHealthChecker{privatefinalJdbcTemplatemasterJdbcTemplate;publicHealthChecker(JdbcTemplatemasterJdbcTemplate){this.masterJdbcTemplate=masterJdbcTemplate;}publicbooleanisMasterDbHealthy(){try{masterJdbcTemplate.queryForObject("SELECT 1",Integer.class);returntrue;}catch(Exceptione){juwatech.cn.util.AsyncLogger.logAsync("Master DB health check failed: "+e.getMessage());returnfalse;}}}演练流程与指标验证
- 启动 Chaos Monkey（
chaos.enabled=true） - 触发高频查券请求
- 监控 Prometheus 指标：
sentinel_block_requests_total（降级次数）http_server_requests_seconds_count{status="500"}（错误率）
- 验证 Kibana 中 fallback 日志是否正常记录
- 模拟主库宕机，观察数据源切换日志与业务影响范围
通过上述机制，系统在节点故障下仍能提供基础服务能力，保障核心链路可用性。
本文著作权归 微赚淘客系统3.0 研发团队，转载请注明出处！