Java使用minio上傳整個目錄下的所有內容

目錄

1、添加相關配置

2、添加依賴

3、實現方法

1??基礎版:

2??優化版(推薦使用):

3??上傳遠程主機上的目錄內容:

4??直接上傳遠程主機中的目錄內容


業務背景:需要需要minio進行上傳指定目錄下所有的內容,具體如下:

1、添加相關配置


# minio 服務
minio:#Minio服務所在地址endpoint: http://192.168.1.20:9000# 默認存儲桶名稱 bucketName: minio-upload#訪問的keyaccessKey: zItdfkAbUVWNkIo#訪問的秘鑰secretKey: JlZp3WTG6PFEckFW7UHXqXSzGu1ICwMXQ6Ep#存儲桶列表名稱bucketNames: data-upload,testdata

配置文件?MinioConfig


import io.minio.MinioClient;
import io.minio.errors.ServerException;
import lombok.Data;
import org.springframework.boot.context.properties.ConfigurationProperties;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;import java.util.List;/*** minio 配置** @author ATB* @date 2025*/
@Data
@Configuration
@ConfigurationProperties(prefix = "minio")
public class MinioConfig {/*** 端點*/private String endpoint;/*** 訪問密鑰*/private String accessKey;/*** 密鑰*/private String secretKey;/*** 存儲桶名稱*/private String bucketName;/*** 存儲桶名稱列表*/private List<String> bucketNames;/*** 注入 Minio 客戶端** @return {@link MinioClient }*/@Beanpublic MinioClient minioClient() throws ServerException {try {return MinioClient.builder().endpoint(endpoint).credentials(accessKey, secretKey).build();} catch (Exception e) {throw new ServerException("-----創建Minio客戶端失敗-----" + e.getMessage() , 202, "未處理");}//return MinioClient.builder()//    .endpoint(endpoint)//    .credentials(accessKey, secretKey)//    .build();}
}

當然也可以不用這個配置類的,需要使用@Value獲取配置參數就行。?

2、添加依賴

        <!-- https://mvnrepository.com/artifact/io.minio/minio --><dependency><groupId>io.minio</groupId><artifactId>minio</artifactId><version>8.5.17</version></dependency><dependency><groupId>com.amazonaws</groupId><artifactId>aws-java-sdk-s3</artifactId><version>1.12.540</version></dependency>

3、實現方法

1??和2??是對本地的目錄進行操作,3??是對遠程機器的目錄進行操作。

1??基礎版:

其中minioConfig.getEndpoint()就是什么配置類中的參數,也可以使用如下方法獲取配置文件的內容,還可以直接寫成固定值都是可以的。

@Value("minio.endpoint")
private String endpoint;
    @Autowiredprivate MinioConfig minioConfig;@Resourceprivate MinioClient minioClient;/*** 上傳目錄** @param bucketName    存儲桶名稱* @param directoryPath directory path (目錄路徑) /home/data/app/models* @return {@link ResponseEntity }<{@link Object }>*/@ApiOperation(value = "上傳目錄")@GetMapping("/uploadCatalog")public ResponseEntity<Object> uploadCatalog(@RequestParam("bucketName") String bucketName, @RequestParam("directoryPath") String directoryPath) {System.out.println("bucketName = " + bucketName);System.out.println("directoryPath = " + directoryPath);try {// 初始化 MinIO 客戶端MinioClient minioClient = MinioClient.builder().endpoint(minioConfig.getEndpoint()).credentials(minioConfig.getAccessKey(), minioConfig.getSecretKey()).build();// 遞歸遍歷目錄及其子目錄Files.walkFileTree(Paths.get(directoryPath), new SimpleFileVisitor<Path>() {@Overridepublic FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) throws IOException {// 將子目錄視為普通文件上傳String relativePath;if (directoryPath.length() + 1 > dir.toString().length()) {// 如果超出范圍,可以選擇返回空字符串或其他默認值relativePath = " /home/data/app/models";} else {relativePath = StringUtils.substringAfter(dir.toString(), directoryPath);}log.info("Uploading directory: " + relativePath);try {// 上傳目錄作為文件minioClient.putObject(PutObjectArgs.builder().bucket(bucketName)// 結尾加斜杠表示目錄.object("/model-file-path/" + relativePath).stream(new FileInputStream(dir.toString()), 0, -1)// 設置為目錄類型.contentType("application/x-directory").build());log.info("Uploaded directory: " + relativePath);} catch (Exception e) {log.error("Failed to upload directory: " + dir.getFileName());e.printStackTrace();}// 繼續遍歷子目錄return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFile(Path file, BasicFileAttributes attrs) throws IOException {// 上傳文件String relativePath = file.toString().substring(directoryPath.length() + 1);log.info("Uploading file: " + relativePath);try {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object("/model-file-path/" + relativePath).stream(new FileInputStream(file.toFile()), Files.size(file), -1)// 自動檢測文件類型.contentType(Files.probeContentType(file)) .build());log.info("Uploaded file: " + relativePath);} catch (Exception e) {log.warn("Failed to upload file: " + file.getFileName());e.printStackTrace();}// 繼續遍歷下一個文件return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFileFailed(Path file, IOException exc) throws IOException {log.warn("Failed to process file: " + file.getFileName() + ", reason: " + exc.getMessage());// 忽略失敗的文件return FileVisitResult.CONTINUE;}});log.info("All files and directories uploaded successfully.");} catch (IOException e) {log.warn("Error occurred: " + e.getMessage());e.printStackTrace();}HashMap<String, Object> response = new HashMap<>();response.put("data", "directoryPath");response.put("code", HttpStatus.OK.value());response.put("msg", "上傳成功");return ResponseEntity.status(HttpStatus.OK).body(response);}

2??優化版(推薦使用):

不同的文件類型也都可以直接上傳

/*** 上傳內容目錄** @param bucketName    存儲桶名稱* @param directoryPath directory path (目錄路徑)* @return {@link ResponseEntity }<{@link Object }>*/@GetMapping("/uploadContentsCatalog")public ResponseEntity<Object> uploadContentsCatalog(@RequestParam("bucketName") String bucketName,@RequestParam("directoryPath") String directoryPath) {log.info("Upload catalog: bucket={}, path={}", bucketName, directoryPath);Map<String, Object> response = new HashMap<>();List<String> successList = new ArrayList<>();List<String> failedList = new ArrayList<>();try {MinioClient minioClient = MinioClient.builder().endpoint(minioConfig.getEndpoint()).credentials(minioConfig.getAccessKey(), minioConfig.getSecretKey()).build();Path basePath = Paths.get(directoryPath);Files.walkFileTree(basePath, new SimpleFileVisitor<Path>() {@Overridepublic FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {if (!dir.equals(basePath)) { // 根目錄不需要作為“目錄對象”上傳String relativePath = basePath.relativize(dir).toString().replace("\\", "/") + "/";try {// “模擬”上傳一個空對象表示目錄(非必要,可跳過)minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object("/model-file-path/" + relativePath).stream(new ByteArrayInputStream(new byte[0]), 0, -1).contentType("application/x-directory").build());log.info("Uploaded directory: {}", relativePath);successList.add(relativePath);} catch (Exception e) {log.error("Failed to upload directory: {}", relativePath, e);failedList.add(relativePath);}}return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {String relativePath = basePath.relativize(file).toString().replace("\\", "/");try (InputStream is = new FileInputStream(file.toFile())) {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object("/model-file-path/" + relativePath).stream(is, Files.size(file), -1)//.contentType(Files.probeContentType(file)).contentType(Optional.ofNullable(Files.probeContentType(file)).orElse("application/octet-stream")).build());log.info("Uploaded file: {}", relativePath);successList.add(relativePath);} catch (Exception e) {log.warn("Failed to upload file: {}", relativePath, e);failedList.add(relativePath);}return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFileFailed(Path file, IOException exc) {String relativePath = basePath.relativize(file).toString().replace("\\", "/");log.warn("Could not access file: {}, reason: {}", relativePath, exc.getMessage());failedList.add(relativePath);return FileVisitResult.CONTINUE;}});} catch (Exception e) {log.error("Unexpected error during directory upload", e);return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(Map.of("code", 500, "msg", "上傳過程中出現錯誤", "error", e.getMessage()));}response.put("code", HttpStatus.OK.value());response.put("msg", "上傳完成");response.put("successCount", successList.size());response.put("failCount", failedList.size());response.put("successItems", successList);response.put("failedItems", failedList);return ResponseEntity.ok(response);}

3??上傳遠程主機上的目錄內容:

實現從遠程主機上傳目錄到 MinIO,你可以通過 SSH(SFTP 或 SCP)下載遠程目錄到本地臨時目錄,然后按之前邏輯上傳至 MinIO。

添加依賴(JSch 連接遠程機器)

<dependency><groupId>com.jcraft</groupId><artifactId>jsch</artifactId><version>0.1.55</version>
</dependency>

具體實現

    /*** 上傳遠程目錄** @param bucketName      存儲桶名稱* @param remoteDirectory remote 目錄* @return {@link ResponseEntity }<{@link Object }>*/@PostMapping("/uploadRemoteCatalog")public ResponseEntity<Object> uploadRemoteCatalog(@RequestParam("bucketName") String bucketName,@RequestParam("remoteDirectory") String remoteDirectory) {String remoteHost = "192.168.1.88";String username = "root";String password = "12345678";String tempDir = System.getProperty("java.io.tmpdir") + "/upload_" + UUID.randomUUID();File localTempDir = new File(tempDir);localTempDir.mkdirs();log.info("Start downloading remote dir [{}] from {} to local temp: {}", remoteDirectory, remoteHost, tempDir);Map<String, Object> response = new HashMap<>();List<String> successList = new ArrayList<>();List<String> failedList = new ArrayList<>();try {// 1. 下載遠程目錄到本地臨時目錄JSch jsch = new JSch();Session session = jsch.getSession(username, remoteHost, 22);session.setPassword(password);session.setConfig("StrictHostKeyChecking", "no");session.connect();Channel channel = session.openChannel("sftp");channel.connect();ChannelSftp sftpChannel = (ChannelSftp) channel;// 下載遠程目錄到本地臨時目錄downloadRemoteDirectory(sftpChannel, remoteDirectory, tempDir);sftpChannel.exit();session.disconnect();log.info("Remote directory downloaded to local temp: {}", tempDir);// 2. 上傳本地目錄到 MinIO(與之前邏輯一致)Path basePath = Paths.get(tempDir);MinioClient minioClient = MinioClient.builder().endpoint(minioConfig.getEndpoint()).credentials(minioConfig.getAccessKey(), minioConfig.getSecretKey()).build();Files.walkFileTree(basePath, new SimpleFileVisitor<Path>() {@Overridepublic FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {if (!dir.equals(basePath)) {String relativePath = basePath.relativize(dir).toString().replace("\\", "/") + "/";try {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object("/model-file-path/" + relativePath).stream(new ByteArrayInputStream(new byte[0]), 0, -1).contentType("application/x-directory").build());successList.add(relativePath);} catch (Exception e) {log.error("Failed to upload directory: {}", relativePath, e);failedList.add(relativePath);}}return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {String relativePath = basePath.relativize(file).toString().replace("\\", "/");try (InputStream is = new FileInputStream(file.toFile())) {String contentType = Optional.ofNullable(Files.probeContentType(file)).orElse("application/octet-stream");minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object("/model-file-path/" + relativePath).stream(is, Files.size(file), -1).contentType(contentType).build());successList.add(relativePath);} catch (Exception e) {log.warn("Failed to upload file: {}", relativePath, e);failedList.add(relativePath);}return FileVisitResult.CONTINUE;}});} catch (Exception e) {log.error("Upload from remote server failed", e);return ResponseEntity.status(HttpStatus.INTERNAL_SERVER_ERROR).body(Map.of("code", 500, "msg", "遠程上傳失敗", "error", e.getMessage()));} finally {// 3. 清理刪除臨時目錄try {log.warn("Delete temp dir: {}", localTempDir);FileUtils.deleteDirectory(localTempDir);} catch (IOException e) {log.warn("Failed to delete temp dir: {}", tempDir);}}response.put("code", HttpStatus.OK.value());response.put("msg", "上傳完成");response.put("successCount", successList.size());response.put("failCount", failedList.size());response.put("successItems", successList);response.put("failedItems", failedList);return ResponseEntity.ok(response);}/*** 下載 Remote Directory** @param sftp      SFTP* @param remoteDir 遠程目錄* @param localDir  本地目錄* @throws SftpException SFTP 異常*/private void downloadRemoteDirectory(ChannelSftp sftp, String remoteDir, String localDir) throws SftpException {File localFolder = new File(localDir);if (!localFolder.exists()) {localFolder.mkdirs();}@SuppressWarnings("unchecked")Vector<ChannelSftp.LsEntry> entries = sftp.ls(remoteDir);for (ChannelSftp.LsEntry entry : entries) {String fileName = entry.getFilename();if (".".equals(fileName) || "..".equals(fileName)) continue;String remotePath = remoteDir + "/" + fileName;String localPath = localDir + "/" + fileName;if (entry.getAttrs().isDir()) {downloadRemoteDirectory(sftp, remotePath, localPath);} else {try (OutputStream output = new FileOutputStream(localPath)) {sftp.get(remotePath, output);} catch (IOException e) {log.error("Failed to download file: {}", remotePath, e);}}}}

至此可以實現遠程上傳到minio中。遠程上傳文件目錄還可以繼續優化,如考慮:支持私鑰登錄、上傳前壓縮、并發、上傳進度等,所以可以形成一個遠程上傳的工具類,具體如下:


import com.jcraft.jsch.*;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import lombok.extern.slf4j.Slf4j;
import org.apache.commons.io.FileUtils;import java.io.*;
import java.nio.file.*;
import java.nio.file.attribute.BasicFileAttributes;
import java.util.*;/*** 遠程上傳器** @author ATB* @date 2025*/
@Slf4j
public class RemoteUploader {public static class SSHConfig {public String host;public int port = 22;public String username;public String password;public SSHConfig(String host, String username, String password) {this.host = host;this.username = username;this.password = password;}}public static class UploadResult {public List<String> success = new ArrayList<>();public List<String> failed = new ArrayList<>();}/*** 1?? 直接上傳** @param sshConfig   SSH 配置* @param remoteDir   遠程目錄* @param bucketName  存儲桶名稱* @param minioClient Minio 客戶端* @return {@link UploadResult }* @throws Exception 例外*/public static UploadResult uploadRemoteDirectoryToMinio(SSHConfig sshConfig,String remoteDir,String bucketName,MinioClient minioClient) throws Exception {// 1. 下載遠程目錄String tempDirPath = System.getProperty("java.io.tmpdir") + "/upload_" + UUID.randomUUID();File tempDir = new File(tempDirPath);tempDir.mkdirs();JSch jsch = new JSch();Session session = jsch.getSession(sshConfig.username, sshConfig.host, sshConfig.port);session.setPassword(sshConfig.password);session.setConfig("StrictHostKeyChecking", "no");session.connect();ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");sftp.connect();downloadRemoteDirectory(sftp, remoteDir, tempDirPath);sftp.disconnect();session.disconnect();// 2. 上傳目錄到 MinIOUploadResult result = uploadLocalDirectoryToMinio(tempDirPath, bucketName, minioClient);// 3. 刪除臨時目錄try {FileUtils.deleteDirectory(tempDir);} catch (IOException e) {log.warn("Failed to delete temp dir: {}", tempDirPath);}return result;}private static void downloadRemoteDirectory(ChannelSftp sftp, String remoteDir, String localDir) throws SftpException {new File(localDir).mkdirs();@SuppressWarnings("unchecked")Vector<ChannelSftp.LsEntry> files = sftp.ls(remoteDir);for (ChannelSftp.LsEntry entry : files) {String name = entry.getFilename();if (".".equals(name) || "..".equals(name)) continue;String remotePath = remoteDir + "/" + name;String localPath = localDir + "/" + name;if (entry.getAttrs().isDir()) {downloadRemoteDirectory(sftp, remotePath, localPath);} else {try (OutputStream os = new FileOutputStream(localPath)) {sftp.get(remotePath, os);} catch (IOException e) {log.error("Failed to download file: {}", remotePath, e);}}}}/*** 2?? 將遠程目錄上傳到壓縮后 Minio** @param sshConfig   SSH 配置* @param remoteDir   遠程目錄* @param bucketName  存儲桶名稱* @param minioClient Minio 客戶端* @return {@link UploadResult }* @throws Exception 例外*/public static UploadResult uploadRemoteDirectoryToMinioWithCompression(SSHConfig sshConfig,String remoteDir,String bucketName,MinioClient minioClient) throws Exception {// 建立 SSH 會話JSch jsch = new JSch();Session session = jsch.getSession(sshConfig.username, sshConfig.host, sshConfig.port);session.setPassword(sshConfig.password);session.setConfig("StrictHostKeyChecking", "no");session.connect();// 打包遠程目錄String remoteTarPath = remoteTarDirectory(session, remoteDir);// SFTP 下載 tar.gzChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");sftp.connect();String localTarPath = downloadRemoteTar(sftp, remoteTarPath);// 可選:清理遠程文件sftp.rm(remoteTarPath);sftp.disconnect();session.disconnect();// 本地解壓String extractPath = extractTarGz(localTarPath);// 上傳解壓后的目錄UploadResult result = uploadLocalDirectoryToMinio(extractPath, bucketName, minioClient);// 清理臨時文件new File(localTarPath).delete();FileUtils.deleteDirectory(new File(extractPath));return result;}/*** 在遠程打包 .tar.gz** @param session   會期* @param remoteDir 遠程目錄* @return {@link String }* @throws JSchException JSCH 例外* @throws IOException   io異常*/private static String remoteTarDirectory(Session session, String remoteDir) throws JSchException, IOException {String parent = remoteDir.endsWith("/") ? remoteDir.substring(0, remoteDir.length() - 1) : remoteDir;String tarName = "/tmp/upload_" + UUID.randomUUID() + ".tar.gz";String command = String.format("tar -czf %s -C %s .", tarName, parent);log.info("Executing remote tar: {}", command);ChannelExec exec = (ChannelExec) session.openChannel("exec");exec.setCommand(command);exec.setInputStream(null);exec.setErrStream(System.err);exec.connect();InputStream input = exec.getInputStream();byte[] tmp = new byte[1024];while (input.read(tmp, 0, 1024) >= 0) {}exec.disconnect();return tarName;}/*** 下載 .tar.gz 文件** @param sftp          SFTP* @param remoteTarPath 遠程 tar 路徑* @return {@link String }* @throws Exception 例外*/private static String downloadRemoteTar(ChannelSftp sftp, String remoteTarPath) throws Exception {String localTarPath = System.getProperty("java.io.tmpdir") + "/" + UUID.randomUUID() + ".tar.gz";try (FileOutputStream fos = new FileOutputStream(localTarPath)) {sftp.get(remoteTarPath, fos);}return localTarPath;}/*** 解壓 .tar.gz** @param tarGzPath tar gz 路徑* @return {@link String }* @throws IOException io異常*/private static String extractTarGz(String tarGzPath) throws IOException {String outputDir = tarGzPath.replace(".tar.gz", "_extracted");Files.createDirectories(Paths.get(outputDir));ProcessBuilder pb = new ProcessBuilder("tar", "-xzf", tarGzPath, "-C", outputDir);Process process = pb.start();try {if (process.waitFor() != 0) throw new RuntimeException("Failed to extract tar.gz");} catch (InterruptedException e) {Thread.currentThread().interrupt();}return outputDir;}/*** 將本地目錄上傳到 Minio** @param localDirPath 本地目錄路徑* @param bucketName   存儲桶名稱* @param minioClient  Minio 客戶端* @return {@link UploadResult }* @throws IOException io異常*/private static UploadResult uploadLocalDirectoryToMinio(String localDirPath,String bucketName,MinioClient minioClient) throws IOException {UploadResult result = new UploadResult();Path basePath = Paths.get(localDirPath);Files.walkFileTree(basePath, new SimpleFileVisitor<Path>() {@Overridepublic FileVisitResult preVisitDirectory(Path dir, BasicFileAttributes attrs) {if (!dir.equals(basePath)) {String relativePath = basePath.relativize(dir).toString().replace("\\", "/") + "/";try {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(relativePath).stream(new ByteArrayInputStream(new byte[0]), 0, -1).contentType("application/x-directory").build());result.success.add(relativePath);} catch (Exception e) {log.warn("Failed to upload directory: {}", relativePath, e);result.failed.add(relativePath);}}return FileVisitResult.CONTINUE;}@Overridepublic FileVisitResult visitFile(Path file, BasicFileAttributes attrs) {String relativePath = basePath.relativize(file).toString().replace("\\", "/");try (InputStream is = new FileInputStream(file.toFile())) {String contentType = Optional.ofNullable(Files.probeContentType(file)).orElse("application/octet-stream");minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(relativePath).stream(is, Files.size(file), -1).contentType(contentType).build());result.success.add(relativePath);} catch (Exception e) {log.warn("Failed to upload file: {}", relativePath, e);result.failed.add(relativePath);}return FileVisitResult.CONTINUE;}});return result;}//==[考慮直接遠程上傳等方式]================================================================================================================// TODO: 考慮直接遠程上傳等方式public void uploadRemoteFileToMinio(SSHConfig sshConfig, String remoteFilePath, String bucketName, String objectName, MinioClient minioClient) throws Exception {// 建立 SSH 會話JSch jsch = new JSch();Session session = jsch.getSession(sshConfig.username, sshConfig.host, sshConfig.port);session.setPassword(sshConfig.password);session.setConfig("StrictHostKeyChecking", "no");session.connect();ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");sftp.connect();try (InputStream remoteStream = sftp.get(remoteFilePath)) {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(remoteStream, -1, PutObjectArgs.MIN_MULTIPART_SIZE)// 可通過文件后綴判斷更具體類型.contentType("application/octet-stream").build());log.info("Uploaded from remote: {} to MinIO: {}", remoteFilePath, objectName);} finally {sftp.disconnect();}}/*** 從遠程主機讀取文件流,上傳到 MinIO** @param sshHost     遠程主機 IP 或域名* @param sshPort     SSH 端口,默認 22* @param username    用戶名* @param password    密碼(或設置為 null 以使用私鑰登錄)* @param privateKey  私鑰路徑(或 null)* @param remotePath  遠程文件絕對路徑* @param minioClient MinIO 客戶端* @param bucketName  MinIO bucket 名稱* @param objectName  上傳后對象名(key)*/public static void uploadRemoteFileToMinio(String sshHost,int sshPort,String username,String password,String privateKey,String remotePath,MinioClient minioClient,String bucketName,String objectName) throws Exception {JSch jsch = new JSch();if (privateKey != null && !privateKey.isBlank()) {jsch.addIdentity(privateKey);}Session session = jsch.getSession(username, sshHost, sshPort);if (password != null && !password.isBlank()) {session.setPassword(password);}session.setConfig("StrictHostKeyChecking", "no");session.connect();ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");sftp.connect();try (InputStream inputStream = sftp.get(remotePath)) {long fileSize = sftp.lstat(remotePath).getSize();minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(inputStream, fileSize, -1).contentType("application/octet-stream").build());log.info("上傳成功: remote={}, object={}", remotePath, objectName);} catch (Exception e) {log.error("上傳失敗: {}", e.getMessage(), e);throw e;} finally {sftp.disconnect();session.disconnect();}}}

實現調用:

/*** 上傳壓縮** @param bucketName 存儲桶名稱* @param remotePath 遠程路徑* @return {@link ResponseEntity }<{@link ? }>*/@PostMapping("/uploadCompressed")public ResponseEntity<?> uploadCompressed(@RequestParam String bucketName,@RequestParam String remotePath) {try {RemoteUploader.SSHConfig ssh = new RemoteUploader.SSHConfig("192.168.1.88", "root", "12345678");MinioClient client = MinioClient.builder().endpoint(minioConfig.getEndpoint()).credentials(minioConfig.getAccessKey(), minioConfig.getSecretKey()).build();// 直接遠程上傳// RemoteUploader.UploadResult result = RemoteUploader.uploadRemoteDirectoryToMinio(ssh, remotePath, bucketName, client);// 壓縮傳輸上傳RemoteUploader.UploadResult result = RemoteUploader.uploadRemoteDirectoryToMinioWithCompression(ssh, remotePath, bucketName, client);return ResponseEntity.ok(Map.of("code", 200,"msg", "上傳完成(壓縮)","successCount", result.success.size(),"failCount", result.failed.size(),"successItems", result.success,"failedItems", result.failed));} catch (Exception e) {return ResponseEntity.status(500).body(Map.of("code", 500,"msg", "上傳失敗","error", e.getMessage()));}}

4??直接上傳遠程主機中的目錄內容

添加工具類:


import com.jcraft.jsch.ChannelSftp;
import com.jcraft.jsch.JSch;
import com.jcraft.jsch.Session;
import com.jcraft.jsch.SftpException;
import io.minio.MinioClient;
import io.minio.PutObjectArgs;
import lombok.extern.slf4j.Slf4j;import java.io.InputStream;
import java.util.ArrayList;
import java.util.List;
import java.util.Vector;
import java.util.concurrent.ExecutorService;
import java.util.concurrent.Executors;
import java.util.concurrent.Future;/*** remote dir uploader** @author ATB* &date  2025/05/28*/
@Slf4j
public class RemoteDirUploader {public static class UploadResult {public List<String> success = new ArrayList<>();public List<String> failed = new ArrayList<>();}/*** 將遠程目錄上傳到 Minio** @param sshHost     SSH 主機* @param sshPort     SSH 端口* @param username    用戶名* @param password    密碼* @param privateKey  私鑰* @param remoteDir   遠程目錄* @param bucketName  存儲桶名稱* @param minioPrefix minio 前綴* @param minioClient Minio 客戶端* @throws Exception 例外*/public static UploadResult uploadRemoteDirectoryToMinio(String sshHost, int sshPort,String username, String password, String privateKey,String remoteDir, String bucketName, String minioPrefix,MinioClient minioClient) throws Exception {JSch jsch = new JSch();if (privateKey != null && !privateKey.isBlank()) {jsch.addIdentity(privateKey);}Session session = jsch.getSession(username, sshHost, sshPort);if (password != null && !password.isBlank()) {session.setPassword(password);}session.setConfig("StrictHostKeyChecking", "no");session.connect();ChannelSftp sftp = (ChannelSftp) session.openChannel("sftp");sftp.connect();UploadResult result;try {result = uploadRecursive(sftp, remoteDir, remoteDir, bucketName, minioPrefix, minioClient);} finally {sftp.disconnect();session.disconnect();}return result;}private static UploadResult uploadRecursive(ChannelSftp sftp, String basePath, String currentPath,String bucketName, String minioPrefix,MinioClient minioClient) throws Exception {UploadResult result = new UploadResult();@SuppressWarnings("unchecked")Vector<ChannelSftp.LsEntry> entries = sftp.ls(currentPath);for (ChannelSftp.LsEntry entry : entries) {String filename = entry.getFilename();if (filename.equals(".") || filename.equals("..")) continue;String fullPath = currentPath + "/" + filename;String relativePath = fullPath.substring(basePath.length()).replaceFirst("^/", "");String objectName = minioPrefix + relativePath;if (entry.getAttrs().isDir()) {// 遞歸子目錄uploadRecursive(sftp, basePath, fullPath, bucketName, minioPrefix, minioClient);} else {long fileSize = entry.getAttrs().getSize();try (InputStream inputStream = sftp.get(fullPath)) {minioClient.putObject(PutObjectArgs.builder().bucket(bucketName).object(objectName).stream(inputStream, fileSize, -1).contentType("application/octet-stream").build());log.info("上傳成功: {}", objectName);result.success.add(objectName);} catch (Exception e) {log.warn("上傳失敗: {} - {}", objectName, e.getMessage());result.failed.add(objectName);}}}return result;}
}

然后接口調用:

/*** 流式直接上傳** @param bucketName 存儲桶名稱* @param remotePath 遠程路徑* @return {@link ResponseEntity }<{@link ? }>*/@PostMapping("/streamingUploads")public ResponseEntity<?> streamingUploads(@RequestParam String bucketName,@RequestParam String remotePath) {try {MinioClient client = MinioClient.builder().endpoint(minioConfig.getEndpoint()).credentials(minioConfig.getAccessKey(), minioConfig.getSecretKey()).build();RemoteDirUploader.UploadResult result = RemoteDirUploader.uploadRemoteDirectoryToMinio("192.168.5.88", 22, "root", "123456", null,remotePath, bucketName, "model-file/data/",minioClient);return ResponseEntity.ok(Map.of("code", 200,"msg", "上傳完成","successCount", result.success.size(),"failCount", result.failed.size(),"successItems", result.success,"failedItems", result.failed));} catch (Exception e) {return ResponseEntity.status(500).body(Map.of("code", 500,"msg", "上傳失敗","error", e.getMessage()));}}

至此!就可以實現本地或者是遠程機器上的目錄上傳到minio中了!

本文來自互聯網用戶投稿,該文觀點僅代表作者本人,不代表本站立場。本站僅提供信息存儲空間服務,不擁有所有權,不承擔相關法律責任。
如若轉載,請注明出處:http://www.pswp.cn/bicheng/82694.shtml
繁體地址,請注明出處:http://hk.pswp.cn/bicheng/82694.shtml
英文地址,請注明出處:http://en.pswp.cn/bicheng/82694.shtml

如若內容造成侵權/違法違規/事實不符,請聯系多彩編程網進行投訴反饋email:809451989@qq.com,一經查實,立即刪除!

相關文章

Python的分布式網絡爬蟲系統實現

1. 系統架構概述 一個典型的分布式網絡爬蟲系統通常包含以下幾個核心組件&#xff1a; 1.主節點&#xff08;Master Node&#xff09;&#xff1a; 任務調度&#xff1a;負責將抓取任務分配給各個工作節點。URL 管理&#xff1a;維護待抓取的 URL 隊列和已抓取的 URL 集合&a…

AI工具的選擇:Dify還是傳統工具?

從純技術視角出發&#xff0c;選擇Dify還是傳統開發工具需要基于六個核心維度進行理性決策。以下為結構化分析框架&#xff0c;附典型場景示例&#xff1a; 1. 開發效率 vs 控制力權衡矩陣 維度Dify優勢場景傳統工具優勢場景迭代速度需求明確的標準CRUD&#xff08;如后臺管理…

2.3 TypeScript 非空斷言操作符(后綴 !)詳解

在 TypeScript 中&#xff0c;當你開啟了嚴格的空值檢查&#xff08;strictNullChecks&#xff09;后&#xff0c;變量如果可能是 null 或 undefined&#xff0c;就必須在使用前進行顯式的判斷。為了在某些場景下簡化代碼&#xff0c;TypeScript 提供了非空斷言操作符&#xff…

深度學習:損失函數與激活函數全解析

目錄 深度學習中常見的損失函數和激活函數詳解引言一、損失函數詳解1.1 損失函數的作用與分類1.2 回歸任務損失函數1.2.1 均方誤差&#xff08;MSE&#xff09;1.2.2 平均絕對誤差&#xff08;MAE&#xff09; 1.3 分類任務損失函數1.3.1 交叉熵損失&#xff08;Cross-Entropy&…

掌握 npm 核心操作:從安裝到管理依賴的完整指南

圖為開發者正在終端操作npm命令&#xff0c;圖片來源&#xff1a;Unsplash 作為 Node.js 生態的基石&#xff0c;npm&#xff08;Node Package Manager&#xff09;是每位開發者必須精通的工具。每天有超過 1700 萬個項目通過 npm 共享代碼&#xff0c;其重要性不言而喻。本文…

Elasticsearch的運維

Elasticsearch 運維工作詳解&#xff1a;從基礎保障到性能優化 Elasticsearch&#xff08;簡稱 ES&#xff09;作為分布式搜索和分析引擎&#xff0c;其運維工作需要兼顧集群穩定性、性能效率及數據安全。以下從核心運維模塊展開說明&#xff0c;結合實踐場景提供可落地的方案…

國產三維CAD皇冠CAD(CrownCAD)建模教程:汽車電池

在線解讀『汽車電池』的三維建模流程&#xff0c;講解3D草圖、保存實體、拉伸凸臺/基體、設置外觀等操作技巧&#xff0c;一起和皇冠CAD&#xff08;CrownCAD&#xff09;學習制作步驟吧&#xff01; 汽車電池&#xff08;通常指鉛酸蓄電池或鋰離子電池&#xff09;是車輛電氣系…

深入理解 JDK、JRE 和 JVM 的區別

在 Java 中&#xff0c;JDK、JRE 和 JVM 是非常重要的概念&#xff0c;它們各自扮演著不同的角色&#xff0c;卻又緊密相連。今天&#xff0c;就讓我們來詳細探討一下它們之間的區別。 一、JVM JVM 即 Java 虛擬機&#xff0c;它是整個 Java 技術體系的核心。JVM 提供了 Java…

云電腦顯卡性能終極對決:ToDesk云電腦/順網云/海馬云,誰才是4K游戲之王?

一、引言 1.1 云電腦的算力革命 云電腦與傳統PC的算力供給差異 傳統PC的算力構建依賴用戶一次性配置本地硬件&#xff0c;特別是CPU與顯卡&#xff08;GPU&#xff09;。而在高性能計算和游戲圖形渲染等任務中&#xff0c;GPU的能力往往成為決定體驗上限的核心因素。隨著游戲分…

撤銷Conda初始化

在安裝miniconda3的過程中&#xff0c;最后系統會出現這一行提示用戶可以選擇自動初始化&#xff0c;這樣的話&#xff0c;系統每次啟動就會自動啟動基礎&#xff08;base&#xff09;環境。 但是我們也可以通過 conda init --reverse $shell 來撤銷 Conda 的初始化設置。這將恢…

Flask-SQLAlchemy數據庫查詢:query

1、為什么可以用 模型類.query 來查詢數據庫&#xff1f; 在 Flask 中使用 SQLAlchemy ORM 時&#xff0c;所有繼承自 db.Model 的模型類都會自動獲得一個 query 屬性。 其本質是 db.session.query(模型類) 的快捷方式&#xff0c;無需顯式操作 db.session。 代碼示例&#…

【免費】【無需登錄/關注】度分秒轉換在線工具

UVE Toolbox 功能概述 這是一個用于地理坐標轉換的在線工具&#xff0c;支持兩種轉換模式&#xff1a; 十進制度 → 度分秒 度分秒 → 十進制度 使用方法 十進制度轉度分秒 在"經度"輸入框中輸入十進制度格式的經度值&#xff08;例如&#xff1a;121.46694&am…

怎么判斷一個Android APP使用了React Native 這個跨端框架

要判斷一個 Android 應用是否使用了 React Native 框架&#xff0c;可以通過以下方法逐步驗證&#xff1a; 一、安裝包結構分析 1. 解壓 APK 將 .apk 文件重命名為 .zip 并解壓&#xff0c;檢查以下特征文件&#xff1a; ? assets/index.android.bundle&#xff1a; React Na…

Pluto實驗報告——基于2ASK的簡易的通信系統

一、實驗目的 1. 熟悉并掌握PLUTO SDR 主動學習模塊的使用&#xff1b; 2.通過matlab 編碼與adalm pluto 相配合達成一個簡易的通信系統&#xff0c;并能 夠傳輸一些較為簡單的信息。 二、實驗原理 2ASK 調制原理&#xff1a; 振幅鍵控是指利用載波的振幅變化來傳遞數字基帶信…

Ubuntu 24-部署FTP和自定義用戶

目錄 一、 安裝 vsftpd 二、創建 FTP 數據目錄 三、創建 FTP 用戶 四、配置 vsftpd 五、重啟 vsftpd 服務 六、增加新用戶腳本 一、 安裝 vsftpd sudo apt update sudo apt install vsftpd -y 二、創建 FTP 數據目錄 sudo mkdir -p /data/ftp sudo chown nobody:nogrou…

MySQL問題:什么是MySQL的中的最左匹配原則?

是指在復合索引中&#xff0c;查詢條件需要按照索引列的順序從最左側列開始依次匹配。只有查詢條件中的列按照索引的最左邊列開始進行匹配&#xff0c;索引才能被有效使用&#xff0c;但有時雖然不是正常順序&#xff0c;由于MySQL中存在優化器&#xff0c;會自動調整順序&…

2025軟考軟件設計師題目

選擇題&#xff08;綜合題&#xff09; 確定得分的 1、Linux外設目錄是什么 /dev。存儲磁盤的目錄 2、Linux外設sdc類型設備屬于什么 scsi hard disk。根據第一個字母s盲猜的 3、計算機中讓程序計數器PC不能指向當前運行程序的技術是 流水線。根據流水線的原理 4、Python程…

Deep Evidential Regression

摘要 翻譯&#xff1a; 確定性神經網絡&#xff08;NNs&#xff09;正日益部署在安全關鍵領域&#xff0c;其中校準良好、魯棒且高效的不確定性度量至關重要。本文提出一種新穎方法&#xff0c;用于訓練非貝葉斯神經網絡以同時估計連續目標值及其關聯證據&#xff0c;從而學習…

每天掌握一個Linux命令 - sqlite3

Linux 命令工具 sqlite3 使用指南 一、工具概述 sqlite3 是 SQLite 數據庫的命令行工具&#xff0c;用于在 Linux 系統中直接操作 SQLite 數據庫&#xff08;輕量級、無服務器、嵌入式關系型數據庫&#xff09;。 核心特點&#xff1a; 無需安裝數據庫服務&#xff0c;直接通…

leetcode:2160. 拆分數位后四位數字的最小和(python3解法,數學相關算法題)

難度&#xff1a;簡單 給你一個四位 正 整數 num 。請你使用 num 中的 數位 &#xff0c;將 num 拆成兩個新的整數 new1 和 new2 。new1 和 new2 中可以有 前導 0 &#xff0c;且 num 中 所有 數位都必須使用。 比方說&#xff0c;給你 num 2932 &#xff0c;你擁有的數位包括…