Java面試必備之JVM+GC教程
這幾天閑著在優銳課的java學習必備中學習了,在本文中,了解如何使用Spring Batch通過StaxEventItemReader使用ItemReader讀取XML文件并將其數據寫入NoSQL。
在本文中,我們將向展示如何使用Spring Batch使用StaxEventItemReader和ItemReader讀取XML文件,以及如何使用帶有JpaRepository的Custom ItemWriter將其數據寫入NoSQL。在這里,我們使用了MongoDB。
自定義ItemReader或ItemWriter是一個類,我們在其中編寫自己的讀取或寫入數據的方式。在Custom Reader中,我們也需要處理分塊邏輯。如果我們的讀取邏輯很復雜并且無法使用spring提供的Default ItemReader進行處理,那么這將很方便。
使用的工具和庫:
1. Maven 3.5+
2. Spring Batch Starter
3. Spring OXM
4. Data Mongodb starter
5. xstream
Maven依賴關系- 需要配置項目。
<?xml version="1.0" encoding="UTF-8"?>
<project xmlns="http://maven.apache.org/POM/4.0.0" xmlns:xsi="http://www.w3.org/2001/XMLSchema-instance"xsi:schemaLocation="http://maven.apache.org/POM/4.0.0 https://maven.apache.org/xsd/maven-4.0.0.xsd"><modelVersion>4.0.0</modelVersion><p><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-parent</artifactId><version>2.2.2.RELEASE</version><relativePath ></relativePath> <!-- lookup parent from repository --></parent><groupId>com.example</groupId><artifactId>spring-batch-mongodb</artifactId><version>0.0.1-SNAPSHOT</version><name>spring-batch-mongodb</name><description>Demo project for Spring Boot</description><p><java.version>1.8</java.version><maven-jar-plugin.version>3.1.1</maven-jar-plugin.version></properties><dependencies><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-batch</artifactId></dependency><dependency><groupId>org.springframework</groupId><artifactId>spring-oxm</artifactId></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-data-mongodb</artifactId></dependency><dependency><groupId>com.thoughtworks.xstream</groupId><artifactId>xstream</artifactId><version>1.4.7</version></dependency><dependency><groupId>org.projectlombok</groupId><artifactId>lombok</artifactId><optional>true</optional></dependency><dependency><groupId>org.springframework.boot</groupId><artifactId>spring-boot-starter-test</artifactId><scope>test</scope><exclusions><exclusion><groupId>org.junit.vintage</groupId><artifactId>junit-vintage-engine</artifactId></exclusion></exclusions></dependency><dependency><groupId>org.springframework.batch</groupId><artifactId>spring-batch-test</artifactId><scope>test</scope></dependency><dependency><groupId>com.h2database</groupId><artifactId>h2</artifactId><scope>runtime</scope></dependency></dependencies><build><p><p><groupId>org.springframework.boot</groupId><artifactId>spring-boot-maven-plugin</artifactId></plugin></plugins></build>
</project>
CustomerWriter — 這是我們創建的自定義寫入器,用于將客戶數據寫入MongoDB。自定義編寫器也提供執行復雜操作的功能。
package com.example.writer;
import java.util.List;
import org.springframework.batch.item.ItemWriter;
import org.springframework.beans.factory.annotation.Autowired;
import com.example.domain.Customer;
import com.example.repository.CustomerRepository;
public class CustomerWriter implements ItemWriter<Customer>{@Autowiredprivate CustomerRepository customerRepository;@Overridepublic void write(List<? extends Customer> customers) throws Exception {customerRepository.saveAll(customers);}
}
CustomerRepository — 這是一個Mongo存儲庫,可與Mongo數據庫進行對話并執行操作以取回數據。
package com.example.repository;
import org.springframework.data.mongodb.repository.MongoRepository;
import com.example.domain.Customer;
public interface CustomerRepository extends MongoRepository<Customer, String>{
}
客戶 —這是一個包含業務數據的Mongo文檔類。
package com.example.domain;
import java.time.LocalDate;
import javax.xml.bind.annotation.XmlRootElement;
import org.springframework.data.annotation.Id;
import org.springframework.data.mongodb.core.mapping.Document;
import org.springframework.data.mongodb.core.mapping.Field;
import lombok.AllArgsConstructor;
import lombok.Builder;
import lombok.Data;
import lombok.NoArgsConstructor;
@AllArgsConstructor
@NoArgsConstructor
@Builder
@Data
@XmlRootElement(name = "Customer")
@Document
public class Customer {@Idprivate Long id;@Fieldprivate String firstName;@Fieldprivate String lastName;@Fieldprivate LocalDate birthdate;
}
CustomerConverter — 我們已經實現了Converter接口。此類用于Converter實現,并且負責將Java對象與文本數據進行編組。如果在處理期間發生異常,則應引發ConversionException。如果使用高級com.thoughtworks.xstream.XStream門面,則可以使用XStream.registerConverter()方法注冊新的轉換器。
package com.example.config;
import java.time.LocalDate;
import java.time.format.DateTimeFormatter;
import com.example.domain.Customer;
import com.thoughtworks.xstream.converters.Converter;
import com.thoughtworks.xstream.converters.MarshallingContext;
import com.thoughtworks.xstream.converters.UnmarshallingContext;
import com.thoughtworks.xstream.io.HierarchicalStreamReader;
import com.thoughtworks.xstream.io.HierarchicalStreamWriter;
public class CustomerConverter implements Converter {private static final DateTimeFormatter DT_FORMATTER = DateTimeFormatter.ofPattern("dd-MM-yyyy HH:mm:ss");@Overridepublic boolean canConvert(Class type) {return type.equals(Customer.class);}@Overridepublic void marshal(Object source, HierarchicalStreamWriter writer, MarshallingContext context) {// Don't do anything}@Overridepublic Object unmarshal(HierarchicalStreamReader reader, UnmarshallingContext context) {reader.moveDown();Customer customer = new Customer();customer.setId(Long.valueOf(reader.getValue()));reader.moveUp();reader.moveDown();customer.setFirstName(reader.getValue());reader.moveUp();reader.moveDown();customer.setLastName(reader.getValue());reader.moveUp();reader.moveDown();customer.setBirthdate(LocalDate.parse(reader.getValue(), DT_FORMATTER));return customer;}
}
JobConfiguration- 這是負責執行批處理作業的主要類。在此類中,我們使用了各種Bean來執行單獨的任務。
StaxEventItemReader — 用于讀取基于StAX的XML輸入的項目讀取器。它從輸入的XML文檔中提取片段,該片段對應于要處理的記錄。片段用StartDocument和EndDocument事件包裝,以便可以像獨立XML文檔一樣對片段進行進一步處理。該實現不是線程安全的。
CustomerWriter —這是一個自定義類,可將數據寫入MongoDB。
step1 —此步驟配置ItemReader和ItemWriter,但是ItemProcessor是可選步驟,我們已跳過。
作業- 代表作業的批處理域對象。Job是一個顯式抽象,表示開發人員指定的作業配置。應當注意,重新啟動策略是整體上應用的,而不是步驟。
package com.example.config;
import java.util.HashMap;
import java.util.Map;
import org.springframework.batch.core.Job;
import org.springframework.batch.core.Step;
import org.springframework.batch.core.configuration.annotation.JobBuilderFactory;
import org.springframework.batch.core.configuration.annotation.StepBuilderFactory;
import org.springframework.batch.item.xml.StaxEventItemReader;
import org.springframework.beans.factory.annotation.Autowired;
import org.springframework.context.annotation.Bean;
import org.springframework.context.annotation.Configuration;
import org.springframework.core.io.ClassPathResource;
import org.springframework.oxm.xstream.XStreamMarshaller;
import com.example.domain.Customer;
import com.example.writer.CustomerWriter;
@Configuration
public class JobConfiguration {@Autowiredprivate JobBuilderFactory jobBuilderFactory;@Autowiredprivate StepBuilderFactory stepBuilderFactory;@Beanpublic StaxEventItemReader<Customer> customerItemReader(){Map<String, Class> aliases = new HashMap<>();aliases.put("customer", Customer.class);CustomerConverter converter = new CustomerConverter();XStreamMarshaller ummarshaller = new XStreamMarshaller();ummarshaller.setAliases(aliases);ummarshaller.setConverters(converter);StaxEventItemReader<Customer> reader = new StaxEventItemReader<>();reader.setResource(new ClassPathResource("/data/customer.xml"));reader.setFragmentRootElementName("customer")reader.setUnmarshaller(ummarshaller);return reader;}@Beanpublic CustomerWriter customerWriter() {return new CustomerWriter();}@Beanpublic Step step1() throws Exception {return stepBuilderFactory.get("step1").<Customer, Customer>chunk(200).reader(customerItemReader()).writer(customerWriter()).build();}@Beanpublic Job job() throws Exception {return jobBuilderFactory.get("job").start(step1()).build();}
}
application.properties
spring.data.mongodb.host=localhost
spring.data.mongodb.port=27017
Customer.xml —這是Spring Batch讀取的示例數據。
<?xml version="1.0" encoding="UTF-8" ?>
<customers><customer><id>1</id><firstName>John</firstName><lastName>Doe</lastName><birthdate>10-10-1988 19:43:23</birthdate></customer><customer><id>2</id><firstName>James</firstName><lastName>Moss</lastName><birthdate>01-04-1991 10:20:23</birthdate></customer><customer><id>3</id><firstName>Jonie</firstName><lastName>Gamble</lastName><birthdate>21-07-1982 11:12:13</birthdate></customer><customer><id>4</id><firstName>Mary</firstName><lastName>Kline</lastName><birthdate>07-08-1973 11:27:42</birthdate></customer><customer><id>5</id><firstName>William</firstName><lastName>Lockhart</lastName><birthdate>04-04-1994 04:15:11</birthdate></customer><customer><id>6</id><firstName>John</firstName><lastName>Doe</lastName><birthdate>10-10-1988 19:43:23</birthdate></customer><customer><id>7</id><firstName>Kristi</firstName><lastName>Dukes</lastName><birthdate>17-09-1983 21:22:23</birthdate></customer><customer><id>8</id><firstName>Angel</firstName><lastName>Porter</lastName><birthdate>15-12-1980 18:09:09</birthdate></customer><customer><id>9</id><firstName>Mary</firstName><lastName>Johnston</lastName><birthdate>07-07-1987 19:43:03</birthdate></customer><customer><id>10</id><firstName>Linda</firstName><lastName>Rodriguez</lastName><birthdate>16-09-1991 09:13:43</birthdate></customer><customer><id>11</id><firstName>Phillip</firstName><lastName>Lopez</lastName><birthdate>18-12-1965 11:10:09</birthdate></customer><customer><id>12</id><firstName>Peter</firstName><lastName>Dixon</lastName><birthdate>09-05-1996 19:09:23</birthdate></customer>
</customers>
MainApp — SpringBatchMongodbApplication可以作為Spring Boot項目運行。
package com.example;
import org.springframework.batch.core.configuration.annotation.EnableBatchProcessing;
import org.springframework.boot.SpringApplication;
import org.springframework.boot.autoconfigure.SpringBootApplication;
import org.springframework.boot.autoconfigure.jdbc.DataSourceAutoConfiguration;
import org.springframework.data.mongodb.repository.config.EnableMongoRepositories;
@SpringBootApplication(exclude = {DataSourceAutoConfiguration.class})
@EnableBatchProcessing
@EnableMongoRepositories(basePackages = "com.example.repository")
public class SpringBatchMongodbApplication {public static void main(String[] args) {SpringApplication.run(SpringBatchMongodbApplication.class, args);}
}
輸出:我們可以得出結論,Spring Batch已經使用建議的模式/文檔類型讀取了數據并將其寫入MongoDB。
db.getCollection('customer').find({})
/* 1 */
{"_id" : NumberLong(1),"firstName" : "John","lastName" : "Doe","birthdate" : ISODate("1988-10-09T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 2 */
{"_id" : NumberLong(2),"firstName" : "James","lastName" : "Moss","birthdate" : ISODate("1991-03-31T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 3 */
{"_id" : NumberLong(3),"firstName" : "Jonie","lastName" : "Gamble","birthdate" : ISODate("1982-07-20T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 4 */
{"_id" : NumberLong(4),"firstName" : "Mary","lastName" : "Kline","birthdate" : ISODate("1973-08-06T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 5 */
{"_id" : NumberLong(5),"firstName" : "William","lastName" : "Lockhart","birthdate" : ISODate("1994-04-03T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 6 */
{"_id" : NumberLong(6),"firstName" : "John","lastName" : "Doe","birthdate" : ISODate("1988-10-09T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 7 */
{"_id" : NumberLong(7),"firstName" : "Kristi","lastName" : "Dukes","birthdate" : ISODate("1983-09-16T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 8 */
{"_id" : NumberLong(8),"firstName" : "Angel","lastName" : "Porter","birthdate" : ISODate("1980-12-14T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 9 */
{"_id" : NumberLong(9),"firstName" : "Mary","lastName" : "Johnston","birthdate" : ISODate("1987-07-06T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 10 */
{"_id" : NumberLong(10),"firstName" : "Linda","lastName" : "Rodriguez","birthdate" : ISODate("1991-09-15T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 11 */
{"_id" : NumberLong(11),"firstName" : "Phillip","lastName" : "Lopez","birthdate" : ISODate("1965-12-17T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
/* 12 */
{"_id" : NumberLong(12),"firstName" : "Peter","lastName" : "Dixon","birthdate" : ISODate("1996-05-08T18:30:00.000Z"),"_class" : "com.example.domain.Customer"
}
> 喜歡這篇文章的可以點個贊,歡迎大家留言評論,記得關注我,每天持續更新技術干貨、職場趣事、海量面試資料等等
> 如果你對java技術很感興趣也可以交流學習,共同學習進步。
> 不要再用"沒有時間“來掩飾自己思想上的懶惰!趁年輕,使勁拼,給未來的自己一個交代
文章寫道這里,歡迎完善交流。最后奉上近期整理出來的一套完整的java架構思維導圖,分享給大家對照知識點參考學習。有更多JVM、Mysql、Tomcat、Spring Boot、Spring Cloud、Zookeeper、Kafka、RabbitMQ、RockerMQ、Redis、ELK、Git等Java干貨
