前言
ThreadPoolTaskExecutor是Spring框架提供的一個線程池實現,它是對Java標準庫中ThreadPoolExecutor的封裝,提供了更便捷的配置和集成方式,特別適合在Spring環境中使用。相關線程池概念見線程&線程池相關
CompletableFuture 是 Java 8 引入的異步編程工具,實現了 Future 和CompletionStage 接口。它不僅提供了異步任務執行能力,還支持強大的函數式編程風格,允許開發者以聲明式方式組合多個異步操作,處理復雜的異步編程場景。相關概念及API使用
功能描述
創建了一個ThreadPoolTaskExecutor的管理類用于監控線程池狀態、動態調整線程池配置,定義線程池注冊為Spring Bean,創建基于分頁查詢和同步的CompletableFuture異步任務,使用自定義的核心線程池提交任務,最終主線程獲取異步結果(也可以引申主線程返回任務執行中,記錄任務ID,主動獲取任務執行結果通知主線程,實現頁面操作非阻塞性)。
代碼示例
用于記錄線程池狀態和調整線程池參數的實體類
package gov.zwfw.iam.uc.threadpoolconfig;import lombok.AllArgsConstructor;
import lombok.Data;@Data
@AllArgsConstructor
public class ThreadPoolStatusPo {private String poolName;private int corePoolSize;private int maxPoolSize;private int currentPoolSize;private int activeCount;private int largestPoolSize;private long taskCount;private long completedTaskCount;private int queueSize;private int queueRemainingCapacity;private int queueCapacity;private int keepAliveSeconds;private String rejectedHandlerType;private String threadNamePrefix;private String queueName;
}
package gov.zwfw.iam.uc.threadpoolconfig;import lombok.AllArgsConstructor;
import lombok.Data;
import lombok.NoArgsConstructor;import java.util.concurrent.RejectedExecutionHandler;/*** 用于動態調整線程池配置的實體類* 包括核心線程數、最大線程數、隊列大小、拒絕策略、線程存活時間*/
@Data
@NoArgsConstructor
@AllArgsConstructor
public class ThreadPoolJudgePo {private int corePoolSize;private int maxPoolSize;private int keepAliveSeconds;private int queueCapacity;private RejectedExecutionHandler rejectedExecutionHandler;
}
用于監控線程池狀態和調整線程池參數的管理類
package gov.zwfw.iam.uc.threadpoolconfig;import org.apache.tomcat.util.threads.TaskQueue;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;
import org.springframework.stereotype.Component;import java.util.HashMap;
import java.util.Map;
import java.util.concurrent.BlockingQueue;
import java.util.concurrent.ConcurrentHashMap;
import java.util.concurrent.ThreadPoolExecutor;@Component
public class ThreadPoolManager {//存儲所有注冊的線程池private static final Map<String, ThreadPoolTaskExecutor> threadPoolMap = new ConcurrentHashMap<>();//存儲線程池原始配置(用于重置)private static final Map<String, ThreadPoolJudgePo> originalConfigMap = new ConcurrentHashMap<>();/*** 注冊線程池* @param poolName* @param threadPoolTaskExecutor* @param threadPoolJudgePo*/public void registerThreadPool(String poolName, ThreadPoolTaskExecutor threadPoolTaskExecutor, ThreadPoolJudgePo threadPoolJudgePo){threadPoolMap.put(poolName,threadPoolTaskExecutor);originalConfigMap.put(poolName, threadPoolJudgePo);}/*** 獲取所有線程池狀態* @return*/public Map<String,ThreadPoolStatusPo> getAllThreadPoolStatus(){Map<String,ThreadPoolStatusPo> statusMap = new HashMap<>();threadPoolMap.forEach((name,executor)->{statusMap.put(name,getThreadPoolStatus(name,executor));});return statusMap;}/*** 獲取單個線程池狀態* @param name* @return*/public ThreadPoolStatusPo getSingleThreadPoolStatus(String name){ThreadPoolTaskExecutor threadPoolTaskExecutor = threadPoolMap.get(name);return getThreadPoolStatus(name,threadPoolTaskExecutor);}/*** 問題:為什么有的屬性從executor(ThreadPoolTaskExecutor)獲取,有的從threadPoolTaskExecutor(ThreadPoolExecutor)獲取?** 原因分析:** ThreadPoolTaskExecutor是Spring對Java原生ThreadPoolExecutor的包裝,它提供了一些額外的配置和功能,同時內部持有一個ThreadPoolExecutor實例。* 線程池的核心狀態(如核心線程數、最大線程數、當前線程數、活躍線程數、歷史最大線程數、任務總數、已完成任務數、隊列大小等)都是ThreadPoolExecutor原生提供的,所以直接從ThreadPoolExecutor實例獲取。* 但是,Spring的ThreadPoolTaskExecutor在配置線程池時,有一些屬性是它自己擴展的,或者需要從它那里獲取配置值,例如:* keepAliveSeconds:在ThreadPoolExecutor中,存活時間是通過getKeepAliveTime(TimeUnit)方法獲取的,但是需要轉換單位。而Spring的ThreadPoolTaskExecutor直接提供了getKeepAliveSeconds()方法,返回的是以秒為單位的值,這樣更方便。* threadNamePrefix:這個前綴是Spring的ThreadPoolTaskExecutor在創建線程工廠時使用的,用于設置線程的名稱前綴,ThreadPoolExecutor本身沒有提供直接獲取線程名稱前綴的方法,所以只能從ThreadPoolTaskExecutor獲取。* 另外,拒絕策略的處理:ThreadPoolExecutor提供了getRejectedExecutionHandler()方法,可以獲取到拒絕策略處理器,然后通過getClass().getName()得到其類名。這里沒有使用Spring的包裝,因為拒絕策略處理器是直接設置在底層的ThreadPoolExecutor上的。* 因此,總結如下:** 大多數運行時狀態(動態的)都是從ThreadPoolExecutor(即threadPoolTaskExecutor)中獲取。* 而一些配置信息,特別是Spring包裝后提供的配置(如keepAliveSeconds和threadNamePrefix)則從ThreadPoolTaskExecutor(即executor)中獲取。* 注意:代碼中有一個屬性是queueCapacity(隊列總容量),它是通過queue.size() + queue.remainingCapacity()計算得到的,因為隊列的剩余容量加上當前已使用的容量就是總容量。** 所以,這樣的設計是合理的,充分利用了Spring的ThreadPoolTaskExecutor提供的便捷方法,同時也直接使用原生的ThreadPoolExecutor來獲取運行時指標。** 但是,這里有一個潛在的問題:Spring的ThreadPoolTaskExecutor的getKeepAliveSeconds()返回的是配置的存活時間(秒),而實際上ThreadPoolExecutor內部是以納秒為單位保存的。不過,由于我們在配置時也是以秒為單位,所以這里獲取的值是一致的。** 另外,關于拒絕策略,這里獲取的是處理器的類名,這樣我們可以知道具體是哪種拒絕策略。* @param name* @param executor* @return*/private ThreadPoolStatusPo getThreadPoolStatus(String name, ThreadPoolTaskExecutor executor) {ThreadPoolExecutor threadPoolTaskExecutor = executor.getThreadPoolExecutor();return new ThreadPoolStatusPo(name,threadPoolTaskExecutor.getCorePoolSize(),threadPoolTaskExecutor.getMaximumPoolSize(),threadPoolTaskExecutor.getPoolSize(),threadPoolTaskExecutor.getActiveCount(),threadPoolTaskExecutor.getLargestPoolSize(),threadPoolTaskExecutor.getTaskCount(),threadPoolTaskExecutor.getCompletedTaskCount(),threadPoolTaskExecutor.getQueue().size(),threadPoolTaskExecutor.getQueue().remainingCapacity(),threadPoolTaskExecutor.getQueue().size() + threadPoolTaskExecutor.getQueue().remainingCapacity(),executor.getKeepAliveSeconds(),threadPoolTaskExecutor.getRejectedExecutionHandler().getClass().getName(),executor.getThreadNamePrefix(),executor.getThreadPoolExecutor().getQueue().getClass().getName());}/*** 動態調整線程池* @param name* @param corePoolSize* @param maxPoolSize* @param queueCapacity*/public void adjustThreadPool(String name,Integer corePoolSize,Integer maxPoolSize,Integer queueCapacity){ThreadPoolTaskExecutor executor = threadPoolMap.get(name);if(null == executor){throw new RuntimeException(name+"線程池不存在");}ThreadPoolExecutor threadPoolExecutor = executor.getThreadPoolExecutor();//調整核心線程數if(null != corePoolSize && corePoolSize > 0){threadPoolExecutor.setCorePoolSize(corePoolSize);}//調整最大線程數if(null != maxPoolSize && maxPoolSize > 0){threadPoolExecutor.setMaximumPoolSize(maxPoolSize);}//調整隊列容量if(null != queueCapacity && queueCapacity > 0){//在Spring的ThreadPoolTaskExecutor中,我們設置隊列容量時,它實際上創建的就是TaskQueue//考慮使用Spring提供的setQueueCapacity方法(通過ThreadPoolTaskExecutor對象),這樣更安全。但是,這個方法會重新設置隊列容量,但不會改變隊列實例,因為內部會調用TaskQueue.setCapacity(如果隊列是TaskQueue的話)//所以,我們可以直接調用executor.setQueueCapacity(queueCapacity)來實現executor.setQueueCapacity(queueCapacity);}}/*** 重置線程池* @param name*/public void resetThreadPool(String name){ThreadPoolJudgePo threadPoolJudgePo = originalConfigMap.get(name);if(null == threadPoolJudgePo){throw new RuntimeException(name+"線程池初始化配置不存在");}adjustThreadPool(name,threadPoolJudgePo.getCorePoolSize(),threadPoolJudgePo.getMaxPoolSize(),threadPoolJudgePo.getQueueCapacity());}}
用于SpringBoot項目注冊線程池Bean的配置類
package gov.zwfw.iam.uc.threadpoolconfig;import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.scheduling.annotation.EnableAsync;
import org.springframework.scheduling.concurrent.ThreadPoolTaskExecutor;import java.util.concurrent.Executor;
import java.util.concurrent.ThreadPoolExecutor;/*** 線程池配置* 用于執行異步任務*/
@Configuration
@EnableAsync
public class ThreadPoolConfig {@AutowiredThreadPoolManager threadPoolManager;/*** 核心線程池配置* @return*/@Bean(name = "coreTaskExecutor")public Executor coreTaskExecutor(){ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();//核心線程數executor.setCorePoolSize(10);//最大線程數executor.setMaxPoolSize(20);//隊列容量,在創建ThreadPoolExecutor時,如果隊列是LinkedBlockingQueue且queueCapacity>0,則將其替換為TaskQueue。executor.setQueueCapacity(500);//空閑線程存活時間executor.setKeepAliveSeconds(60);//拒絕策略,使用調用者運行策略,也可以自定義策略以增強可用性,這里只是簡單推送人員信息,量不是特別大,沒必要費勁executor.setRejectedExecutionHandler(new ThreadPoolExecutor.CallerRunsPolicy());//線程名前綴executor.setThreadNamePrefix("coreTaskExecutor-");//優雅停機配置//等待所有任務結束后再關閉線程池executor.setWaitForTasksToCompleteOnShutdown(true);//等待終止時間executor.setAwaitTerminationSeconds(60);executor.initialize();//注冊到監控threadPoolManager.registerThreadPool("coreTaskExecutor",executor,new ThreadPoolJudgePo(executor.getCorePoolSize(),executor.getMaxPoolSize(),executor.getKeepAliveSeconds(),executor.getThreadPoolExecutor().getQueue().size(),executor.getThreadPoolExecutor().getRejectedExecutionHandler()));System.out.println("=================================="+executor.getThreadPoolExecutor().getQueue().getClass().getName());return executor;}@Bean(name = "commonTaskExecutor")public Executor commonTaskExecutor(){ThreadPoolTaskExecutor executor = new ThreadPoolTaskExecutor();//核心線程數executor.setCorePoolSize(5);//最大線程數executor.setMaxPoolSize(10);//隊列容量executor.setQueueCapacity(100);//空閑線程存活時間executor.setKeepAliveSeconds(120);//拒絕策略,拋棄策略executor.setRejectedExecutionHandler(new ThreadPoolExecutor.AbortPolicy());executor.setThreadNamePrefix("commonTaskExecutor-");executor.initialize();return executor;}
}
用于獲取線程池狀態和調整線程池配置的控制類
package gov.zwfw.iam.uc.threadpoolconfig;import com.alibaba.fastjson.JSONObject;
import gov.zwfw.iam.base.util.StringUtils;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.web.bind.annotation.RequestMapping;
import org.springframework.web.bind.annotation.RequestMethod;
import org.springframework.web.bind.annotation.RequestParam;
import org.springframework.web.bind.annotation.RestController;import java.util.ArrayList;
import java.util.List;
import java.util.Map;
import java.util.stream.Collectors;@RestController
@RequestMapping("/pool")
public class ThreadPoolManageController {@AutowiredThreadPoolManager threadPoolManager;@RequestMapping(value = "/getStatus",method = RequestMethod.GET)public String getStatus(@RequestParam(value = "name",required = false) String name){if(StringUtils.isEmpty(name)){List<String>result = new ArrayList<>();Map<String, ThreadPoolStatusPo> allThreadPoolStatus = threadPoolManager.getAllThreadPoolStatus();allThreadPoolStatus.keySet().forEach(key->{ThreadPoolStatusPo threadPoolStatusPo = allThreadPoolStatus.get(key);String s = key+":"+JSONObject.toJSONString(threadPoolStatusPo);result.add(s);});return result.stream().collect(Collectors.joining("\n"));}else{ThreadPoolStatusPo singleThreadPoolStatus = threadPoolManager.getSingleThreadPoolStatus(name);return name+":"+JSONObject.toJSONString(singleThreadPoolStatus);}}@RequestMapping(value = "/adjust",method = RequestMethod.POST)public String adjust(@RequestParam(value = "name") String name,@RequestParam(value = "corePoolSize",required = false)Integer corePoolSize,@RequestParam(value = "maxPoolSize",required = false)Integer maxPoolSize,@RequestParam(value = "queueCapacity",required = false)Integer queueCapacity){threadPoolManager.adjustThreadPool(name,corePoolSize,maxPoolSize,queueCapacity);return "調整成功";}@RequestMapping(value = "/reset",method = RequestMethod.POST)public String reset(@RequestParam(value = "name") String name){threadPoolManager.resetThreadPool(name);return "重置成功";}
}
提交異步任務的業務實現,這里是分頁查詢用戶信息同步到第三方平臺,首先是頁面點擊實現的全量同步和失敗重試接口
@Value("${batch.push.size:5}")private int batchSize;@RequestMapping("/syncUser")@ResponseBodypublic Result syncUser(@RequestParam String resId) {Result result = new Result();//校驗當天是否存在失敗未重試任務String key = "FAIL_INDEX_"+resId+"_"+new SimpleDateFormat("yyyyMMdd").format(new Date());if(CodisUtil.lLen(key)>0){result.setCode("-5");result.setMsg("存在失敗未重試任務,請點擊失敗重試按鈕處理");return result;}String resUrl = "";try{resUrl = staffApi.selectUserSyncUrl(resId);if(gov.zwfw.iam.base.util.StringUtils.isEmpty(resUrl)){result.setCode("-1");result.setMsg("同步地址為空");return result;}logger.info("同步地址:{}", resUrl);//分頁查詢用戶信息,分批推送int userNum = staffApi.countUser();long startTime = System.nanoTime();CompletableFuture<JSONObject> jsonObject = staffApi.resolveTask(resId,resUrl,userNum,batchSize, null);JSONObject futureJson = jsonObject.get();long endTime = System.nanoTime();logger.info("同步耗時:{}納秒", (endTime-startTime));result.setCode(futureJson.getString("code"));result.setMsg(futureJson.getString("msg"));}catch (Exception e){logger.error("同步失敗,同步地址:{},失敗原因:{}", resUrl, e.getMessage());result.setCode("-1");result.setMsg(e.getMessage());}return result;}@RequestMapping("/syncUserFail")@ResponseBodypublic Result syncUserFail(@RequestParam String resId) {Result result = new Result();String key = "FAIL_INDEX_"+resId+"_"+new SimpleDateFormat("yyyyMMdd").format(new Date());if(CodisUtil.lLen(key)>0){String resUrl = "";try{resUrl = staffApi.selectUserSyncUrl(resId);if(gov.zwfw.iam.base.util.StringUtils.isEmpty(resUrl)){result.setCode("-1");result.setMsg("同步地址為空");return result;}logger.info("同步地址:{}", resUrl);List<String> failIndexList = CodisUtil.lRange(key,0,-1);CodisUtil.delKey(key);long startTime = System.nanoTime();CompletableFuture<JSONObject> jsonObject = staffApi.resolveTask(resId,resUrl,0,batchSize,failIndexList);JSONObject futureJson = jsonObject.get();long endTime = System.nanoTime();logger.info("同步耗時:{}納秒", (endTime-startTime));result.setCode(futureJson.getString("code"));result.setMsg(futureJson.getString("msg"));}catch (Exception e){logger.error("同步失敗,同步地址:{},失敗原因:{}", resUrl, e.getMessage());result.setCode("-1");result.setMsg(e.getMessage());}}else{result.setCode("-6");result.setMsg("不存在失敗未重試任務");}return result;}
然后是真正實現分頁任務以及提交執行的核心類
@Resource(name = "coreTaskExecutor")private Executor coreTaskExecutor;@Async("coreTaskExecutor")public CompletableFuture<JSONObject> resolveTask(String resId, String resUrl, int total, int batchSize, List<String> failIndexList) {//1、分頁任務列表List<CompletableFuture<String>> futures = new ArrayList<>();//失敗的下表存儲到redis的key,使用list類型String key = "FAIL_INDEX_"+resId+"_"+new SimpleDateFormat("yyyyMMdd").format(new Date());//2、如果是全量推送,計算起始位置,執行分頁查詢并推送;如果是失敗數據推送,那么使用失敗的下表,分頁查詢并推送if(total !=0 && null == failIndexList){for(int i=0;i<total;i+=batchSize){int startIndex = i;CompletableFuture<String> future = getFuture(startIndex, batchSize, resUrl, key);futures.add(future);}}else{for (int i = 0; i < failIndexList.size(); i++) {int startIndex = Integer.parseInt(failIndexList.get(i));CompletableFuture<String> future = getFuture(startIndex, batchSize, resUrl, key);futures.add(future);}}//5、等待所有任務執行完成并處理結果CompletableFuture<Void> allFutures = CompletableFuture.allOf(futures.toArray(new CompletableFuture[0]));return allFutures.thenApply(r->{List<String> results = futures.stream().map(CompletableFuture::join).collect(Collectors.toList());//6、構建響應信息JSONObject resultJson = new JSONObject();int failCount = Math.toIntExact(results.stream().filter(result ->{JSONObject jsonObject = JSONObject.parseObject(result);if(jsonObject.containsKey("startIndex")){String startIndex = jsonObject.getString("startIndex");CodisUtil.lPush(key,startIndex);logger.error("失敗index:{}",startIndex);}return !jsonObject.getString("code").equals("0");}).count());resultJson.put("code",failCount>0?"-1":"0");resultJson.put("msg",failCount>0?"部分數據推送失敗,請點擊失敗重試按鈕重新推送":"推送成功");CodisUtil.expireKey(key,60*60*24);return resultJson;});}public int countUser() {return govStaffService.countUser();}public CompletableFuture<String> getFuture(int startIndex, int batchSize, String resUrl, String key){CompletableFuture<String> future = CompletableFuture.supplyAsync(() -> {try{//3、分頁查詢List<UcGovStaff> list = govStaffService.selectListByPage(startIndex, batchSize);logger.info("查詢到第"+(startIndex/batchSize+1)+"頁數據,數量為:{}", list.size());String syncRes = "";if(null == list || list.isEmpty()){JSONObject jsonObject = new JSONObject();jsonObject.put("code","-2");jsonObject.put("msg","推送數據為空");return jsonObject.toJSONString();}//4、執行推送任務syncRes = govStaffService.syncUser(startIndex,list, resUrl);return syncRes;}catch (Exception e){logger.error("分頁任務異常:{}",e.getMessage());JSONObject jsonObject = new JSONObject();jsonObject.put("code","-3");jsonObject.put("msg","任務執行失敗");CodisUtil.lPush(key,String.valueOf(startIndex));CodisUtil.expireKey(key,60*60*24);return jsonObject.toJSONString();}},coreTaskExecutor);return future;}public String syncUser(int startIndex, List<UcGovStaff> list, String resUrl) {String data = JSON.toJSONString(list);JSONObject jsonObject = new JSONObject();jsonObject.put("data", data);String s = "";try {s = WebUtils.doPost(resUrl,jsonObject);jsonObject = JSONObject.parseObject(s);if(!"0".equals(jsonObject.getString("code"))){jsonObject.put("startIndex",startIndex);}s = jsonObject.toJSONString();} catch (IOException e) {logger.error("同步人員異常:{}",e.getMessage());jsonObject = new JSONObject();jsonObject.put("code","-1");jsonObject.put("msg","網絡請求異常");jsonObject.put("startIndex",startIndex);s = jsonObject.toJSONString();}return s;}
注意,使用異步任務一定要在啟動類添加@EnableAsync
注解,同時,真正執行異步任務的方法上添加@Async("coreTaskExecutor")
注解,注解里的參數對應的是提交任務的線程池名稱。下面是獲取線程池狀態以及調整線程池配置的示例
總結
這次業務實現了基于ThreadPoolTaskExecutor+CompletableFuture的數據推送業務,可以引申為其他的多線程異步任務實現,實現了全量數據推送和失敗重試機制,對于處理大批量任務很有幫助,由于業務中主線程和異步任務是同步實現的,因此,會阻塞主線程直至異步任務執行完成,如果要實現主線程同步返回,異步執行后續任務,只需要@Async
注解提交resolveTask任務即可。