文章目錄
- 創建域
- 1.創建域
- 2.輸入配置
- 部署選項
- 數據節點
- 網絡
- 精細訪問控制
- 訪問策略
- 獲取域端點
- 數據如何插入到OpenSearch Service
- Java連接OpenSearch Service
- spring-data-opensearch
- elasticsearch-rest-high-level-client
- opensearch-rest-client
- opensearch-java
因為是開發測試使用,所以選的都是低配,單機,便宜的配置。
地址:https://aws.amazon.com/cn/opensearch-service/
價格:https://aws.amazon.com/cn/opensearch-service/pricing/
- 實例小時數 +
- 所需的存儲量 +
- 傳入和傳出 Amazon OpenSearch Service 的數據
文檔:https://docs.aws.amazon.com/zh_cn/opensearch-service/
創建域
一般在15-30分鐘內創建一個OpenSearch集群,所以請尿完尿再來搞,別憋壞了。
1.創建域
2.輸入配置
-
域名:后面用java等客戶端連接時會在URL中顯示。
-
域創建方法:請選擇標準創建。
- 輕松創建:快速創建 OpenSearch 域以實現高可用性 含備用節點的多可用區,我們要自定義標準創建便宜點。可以理解為一個產線的標準套餐。
- 標準創建:就是自選套餐。
-
模板:選擇開發/測試。
- 部署選項:選擇不含備用節點的域,主打的就是便宜能用就行。
- 可用區:選擇一個可用區。
- 版本:請選擇最新版本。
部署選項
數據節點
- 實例類型:默認是內存優化 - r6g.large.search,2C16G 價格為USD 0.167/H,下面列幾個便宜的。
- t3.small.search 2C2G USD 0.036 ,自己用來個最便宜的,又不是不能用。
- t3.medium.search 2C4G USD 0.073
- 節點數:1個就行。
- 存儲大小:40G就行。
忽略其他冷熱數據存儲,專用主節點,快照配置,以及自定義終端節點部分。
網絡
- 網絡:選擇 Public access(公有訪問權限)
- 開發測試環境,選擇公共訪問權限,生成用VPC訪問
精細訪問控制
啟用
細粒度訪問控制復選框。選擇創建主用戶輸入用戶名和密碼,用戶名和密碼很關鍵,后面訪問dashboard和遠程客戶端連接都要。
忽略 SAML 身份驗證和 Amazon Cognito 身份驗證
訪問策略
選擇 Only use fine-grained access control(僅使用精細訪問控制)。
忽略其余設置,然后選擇 Create(創建)。新域初始化過程通常需要 15-30 分鐘,但可能需要更長的時間,具體取決于配置。
獲取域端點
點擊創建后,通常需要 15-30 分鐘,然后從控制臺 -> 域 -> 一般信息 獲取域端點。
這里給了IPV6和IPV4的dashboard地址和域端點地址。
數據如何插入到OpenSearch Service
- 對于大規模數據,我們建議使用 Amazon Kinesis Data Firehose,這是一項完全托管的服務,可以自動擴展以匹配您的數據吞吐量,并且不需要進行持續的管理。它還可以在加載數據前對其進行轉換、批處理和壓縮。
- Amazon OpenSearch Service 支持與 Logstash 的集成。您可以將 Amazon OpenSearch Service 域配置為數據存儲,用于存儲所有來自 Logstash 的日志。
- 可以使用索引 API 和批量 API 等原生 Elasticsearch(7.10 及更低版本)或 OpenSearch API 將數據加載到域中。
Java連接OpenSearch Service
當使用 Java 連接 OpenSearch 時,有幾個選擇,包括 Spring Data Elasticsearch/opensearch、原生 Elasticsearch 和 OpenSearch 客戶端。下面是每種方式的簡要介紹:
- spring-data-xxx
- 原生Elasticsearch
- OpenSearch
建議基本的CRUD用
spring-data-xxx
復雜的查詢用低級client,如
opensearch-java
或
RestClient lowLevelClient = highLevelClient.getLowLevelClient();
lowLevelClient . lowLevelClient.performRequest()
驗證分為用戶名和密碼進行身份驗證和AWS訪問密鑰(Access Key和Secret Key)
我們當前選的精細控制用的是用戶名密碼驗證,后面生產建議改為IAM驗證。
spring-data-opensearch
-
Github:https://github.com/opensearch-project/spring-data-opensearch
-
官方示例:https://github.com/opensearch-project/spring-data-opensearch/tree/main/spring-data-opensearch-examples
Spring Boot 3.x
1.添加依賴。
<dependency><groupId>org.opensearch.client</groupId><artifactId>spring-data-opensearch-starter</artifactId><version>1.3.0</version>
</dependency>
2.啟動類去除ElasticsearchDataAutoConfiguration
自動配置。
@SpringBootApplication(exclude = {ElasticsearchDataAutoConfiguration.class})
public class OpenSearchDemoApplication {public static void main(String[] args) {SpringApplication.run(OpenSearchDemoApplication.class, args);}
}
3.application.yml增加配置
opensearch:uris: https://localhost:9200username: adminpassword: adminspring:jackson:serialization:INDENT_OUTPUT: true
4.新增model
import java.math.BigDecimal;
import org.springframework.data.annotation.Id;
import org.springframework.data.elasticsearch.annotations.Document;
import org.springframework.data.elasticsearch.annotations.Field;
import org.springframework.data.elasticsearch.annotations.FieldType;@Document(indexName = "marketplace")
public class Product {@Idprivate String id;@Field(type = FieldType.Text, name = "name")private String name;@Field(type = FieldType.Double, name = "price")private BigDecimal price;@Field(type = FieldType.Integer, name = "quantity")private Integer quantity;@Field(type = FieldType.Text, name = "description")private String description;@Field(type = FieldType.Keyword, name = "vendor")private String vendor;
5.新增Repository
import java.math.BigDecimal;
import java.util.List;
import org.opensearch.data.example.model.Product;
import org.springframework.data.elasticsearch.repository.ElasticsearchRepository;
import org.springframework.stereotype.Repository;@Repository
public interface MarketplaceRepository extends ElasticsearchRepository<Product, String> {List<Product> findByNameLikeAndPriceGreaterThan(String name, BigDecimal price);
}
6.新增service
@Service
public class MarketplaceService {private final MarketplaceRepository repository;public MyService(MarketplaceRepository repository) {this.repository = repository;}public void doWork() {repository.deleteAll();Product product = new Product();product.setName("xxx");product.setxxx("xxx");repository.save(product);List<Product> results = repository.findByNameLikeAndPriceGreaterThan("Gierke",xx);}
}
我們還可以通過RestHighLevelClient
獲得lowLevelRest()
客戶端。
@AutowiredRestHighLevelClient highLevelClient;RestClient lowLevelClient = highLevelClient.getLowLevelClient();IndexRequest request = new IndexRequest("spring-data").id(randomID()).source(singletonMap("feature", "high-level-rest-client")).setRefreshPolicy(IMMEDIATE);
IndexResponse response = highLevelClient.index(request,RequestOptions.DEFAULT);// 構建請求Request request = new Request("GET", "/your-index-name/_search");// 添加請求參數,如果需要的話request.addParameter("q", "field:value");// 執行請求Response response = lowLevelClient.performRequest(request);// 處理響應int statusCode = response.getStatusLine().getStatusCode();String responseBody = EntityUtils.toString(response.getEntity());System.out.println("Status Code: " + statusCode);System.out.println("Response Body: " + responseBody);
elasticsearch-rest-high-level-client
1.添加依賴
<dependencies><dependency><groupId>org.elasticsearch.client</groupId><artifactId>elasticsearch-rest-high-level-client</artifactId><version>7.10.2</version> <!-- 替換為你使用的OpenSearch版本 --></dependency>
</dependencies>
2.示例代碼
import org.elasticsearch.action.index.IndexRequest;
import org.elasticsearch.action.index.IndexResponse;
import org.elasticsearch.action.search.SearchRequest;
import org.elasticsearch.action.search.SearchResponse;
import org.elasticsearch.client.RequestOptions;
import org.elasticsearch.client.RestClient;
import org.elasticsearch.client.RestHighLevelClient;
import org.elasticsearch.common.xcontent.XContentType;
import org.elasticsearch.index.query.QueryBuilders;
import org.elasticsearch.search.builder.SearchSourceBuilder;import java.io.IOException;
import java.util.HashMap;
import java.util.Map;public class OpenSearchExample {public static void main(String[] args) {// OpenSearch連接配置String hostname = "your_opensearch_host"; // 替換為你的OpenSearch服務器主機名或IP地址int port = 9200; // 替換為你的OpenSearch服務器端口String scheme = "https"; // 如果使用HTTPS,否則使用"http"String username = "laker";String password = "lakerpwd";try (RestHighLevelClient client = new RestHighLevelClient(RestClient.builder(hostname).setPort(port).setScheme(scheme).setHttpClientConfigCallback(httpClientBuilder ->httpClientBuilder.setDefaultCredentialsProvider(() ->new org.apache.http.auth.UsernamePasswordCredentials(username, password))))) {// 上傳數據Map<String, Object> jsonMap = new HashMap<>();jsonMap.put("field1", "value1");jsonMap.put("field2", "value2");IndexRequest indexRequest = new IndexRequest("your_index").id("your_document_id") // 替換為文檔的ID.source(jsonMap, XContentType.JSON);IndexResponse indexResponse = client.index(indexRequest, RequestOptions.DEFAULT);System.out.println("Index created with ID: " + indexResponse.getId());// 查詢數據SearchRequest searchRequest = new SearchRequest("your_index");SearchSourceBuilder searchSourceBuilder = new SearchSourceBuilder();searchSourceBuilder.query(QueryBuilders.matchAllQuery());searchRequest.source(searchSourceBuilder);SearchResponse searchResponse = client.search(searchRequest, RequestOptions.DEFAULT);// 處理查詢結果System.out.println("Search hits: " + searchResponse.getHits().getTotalHits());} catch (IOException e) {e.printStackTrace();}}
}
opensearch-rest-client
- 文檔:https://opensearch.org/docs/latest/clients/java-rest-high-level/
1.添加依賴
<dependency><groupId>org.opensearch.client</groupId><artifactId>opensearch-rest-high-level-client</artifactId><version>2.11.1</version></dependency>
2.示例代碼
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.apache.http.impl.nio.client.HttpAsyncClientBuilder;
import org.opensearch.action.admin.indices.delete.DeleteIndexRequest;
import org.opensearch.action.delete.DeleteRequest;
import org.opensearch.action.delete.DeleteResponse;
import org.opensearch.action.get.GetRequest;
import org.opensearch.action.get.GetResponse;
import org.opensearch.action.index.IndexRequest;
import org.opensearch.action.index.IndexResponse;
import org.opensearch.action.support.master.AcknowledgedResponse;
import org.opensearch.client.RequestOptions;
import org.opensearch.client.RestClient;
import org.opensearch.client.RestClientBuilder;
import org.opensearch.client.RestHighLevelClient;
import org.opensearch.client.indices.CreateIndexRequest;
import org.opensearch.client.indices.CreateIndexResponse;
import org.opensearch.common.settings.Settings;import java.io.IOException;
import java.util.HashMap;public class RESTClientSample {public static void main(String[] args) throws IOException {final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("lakertest", "xxxx"));//Create a client.RestClientBuilder builder = RestClient.builder(new HttpHost("search-laker-search-xxxx.us-east-2.es.amazonaws.com", 443, "https")).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider));RestHighLevelClient client = new RestHighLevelClient(builder);//Create a non-default index with custom settings and mappings.CreateIndexRequest createIndexRequest = new CreateIndexRequest("custom-index");createIndexRequest.settings(Settings.builder() //Specify in the settings how many shards you want in the index..put("index.number_of_shards", 4).put("index.number_of_replicas", 3));//Create a set of maps for the index's mappings.HashMap<String, String> typeMapping = new HashMap<String,String>();typeMapping.put("type", "integer");HashMap<String, Object> ageMapping = new HashMap<String, Object>();ageMapping.put("age", typeMapping);HashMap<String, Object> mapping = new HashMap<String, Object>();mapping.put("properties", ageMapping);createIndexRequest.mapping(mapping);CreateIndexResponse createIndexResponse = client.indices().create(createIndexRequest, RequestOptions.DEFAULT);//Adding data to the index.IndexRequest request = new IndexRequest("custom-index"); //Add a document to the custom-index we created.request.id("1"); //Assign an ID to the document.HashMap<String, String> stringMapping = new HashMap<String, String>();stringMapping.put("message:", "Testing Java REST client");request.source(stringMapping); //Place your content into the index's source.IndexResponse indexResponse = client.index(request, RequestOptions.DEFAULT);// 查詢GetRequest getRequest = new GetRequest("custom-index", "1");GetResponse response = client.get(getRequest, RequestOptions.DEFAULT);System.out.println(response.getSourceAsString());// 刪除文檔DeleteRequest deleteDocumentRequest = new DeleteRequest("custom-index", "1"); //Index name followed by the ID.DeleteResponse deleteResponse = client.delete(deleteDocumentRequest, RequestOptions.DEFAULT);// 刪除索引DeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest("custom-index"); //Index name.AcknowledgedResponse deleteIndexResponse = client.indices().delete(deleteIndexRequest, RequestOptions.DEFAULT);client.close();}
}
opensearch-java
- 文檔:https://opensearch.org/docs/latest/clients/java/
- 示例代碼:https://github.com/opensearch-project/opensearch-java/tree/main/samples
- 里面很詳細,包含knn部分。
<dependency><groupId>org.opensearch.client</groupId><artifactId>opensearch-java</artifactId><version>2.8.1</version>
</dependency>
@Data
public class IndexData {private String title;private String text;public IndexData() {}public IndexData(String title, String text) {this.title = title;this.text = text;}
}
import org.apache.http.HttpHost;
import org.apache.http.auth.AuthScope;
import org.apache.http.auth.UsernamePasswordCredentials;
import org.apache.http.client.CredentialsProvider;
import org.apache.http.impl.client.BasicCredentialsProvider;
import org.opensearch.client.RestClient;
import org.opensearch.client.json.jackson.JacksonJsonpMapper;
import org.opensearch.client.opensearch.OpenSearchClient;
import org.opensearch.client.opensearch._types.mapping.IntegerNumberProperty;
import org.opensearch.client.opensearch._types.mapping.Property;
import org.opensearch.client.opensearch._types.mapping.TypeMapping;
import org.opensearch.client.opensearch.core.IndexRequest;
import org.opensearch.client.opensearch.core.SearchResponse;
import org.opensearch.client.opensearch.indices.CreateIndexRequest;
import org.opensearch.client.opensearch.indices.DeleteIndexRequest;
import org.opensearch.client.opensearch.indices.IndexSettings;
import org.opensearch.client.transport.OpenSearchTransport;
import org.opensearch.client.transport.rest_client.RestClientTransport;import java.io.IOException;public class OpenSearchClientExample {public static void main(String[] args) throws IOException {final CredentialsProvider credentialsProvider = new BasicCredentialsProvider();credentialsProvider.setCredentials(AuthScope.ANY,new UsernamePasswordCredentials("admin", "admin"));// 私有 opensearchRestClient restClient = RestClient.builder(new HttpHost("localhost", 9200, "https")).setHttpClientConfigCallback(httpClientBuilder -> httpClientBuilder.setDefaultCredentialsProvider(credentialsProvider)).build();OpenSearchTransport transport = new RestClientTransport(restClient, new JacksonJsonpMapper());OpenSearchClient client = new OpenSearchClient(transport);//Create the indexString indexName = "sample-index";//Add some settings to the indexIndexSettings settings = new IndexSettings.Builder().numberOfShards("2").numberOfReplicas("1").build();TypeMapping mapping = new TypeMapping.Builder().properties("age",new Property.Builder().integer(new IntegerNumberProperty.Builder().build()).build()).build();CreateIndexRequest createIndexRequest = new CreateIndexRequest.Builder().index(indexName).settings(settings).mappings(mapping).build();client.indices().create(createIndexRequest);//Index some dataIndexData indexData = new IndexData("Document 1", "Text for document 1");IndexRequest<IndexData> indexRequest = new IndexRequest.Builder<IndexData>().index(indexName).id("1").document(indexData).build();client.index(indexRequest);//Search for the documentSearchResponse<IndexData> searchResponse = client.search(s -> s.index(indexName), IndexData.class);for (int i = 0; i < searchResponse.hits().hits().size(); i++) {System.out.println(searchResponse.hits().hits().get(i).source());}//Delete the documentclient.delete(b -> b.index(indexName).id("1"));// Delete the indexDeleteIndexRequest deleteIndexRequest = new DeleteIndexRequest.Builder().index(indexName).build();client.indices().delete(deleteIndexRequest);try {if (restClient != null) {restClient.close();}} catch (IOException e) {System.out.println(e.toString());}}
}