磁盤達到閾值導致ES無法刪除數據

method [POST], host [http://xx.xxx.xxx.xxx:9200], URI [/security_event/_delete_by_query?slices=1&requests_per_second=-1&ignore_unavailable=false&expand_wildcards=open&allow_no_indices=true&ignore_throttled=true&refresh=true&conflicts=proceed&wait_for_completion=true&timeout=1m], status line [HTTP/1.1 403 Forbidden]\n{\"took\":19,\"timed_out\":false,\"total\":8,\"deleted\":0,\"batches\":1,\"version_conflicts\":0,\"noops\":0,\"retries\":{\"bulk\":0,\"search\":0},\"throttled_millis\":0,\"requests_per_second\":-1.0,\"throttled_until_millis\":0,\"failures\":[{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"ecb098ef-2e3a-4c7a-a282-4484cabb362f\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"30c1da71-25c5-4e1f-a58f-95f6f5abfc52\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"687e14ba-0bc8-466a-83b1-294a3f9b2422\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"e0f25c75-cce3-4c44-9691-d4c79ecb72e1\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"dbfabe19-3318-420c-a7f0-eb74eab25d43\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"222fcf5e-8b72-4bd8-b3c8-a8c615db2ae9\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"690f57a2-1306-4749-8dfc-a1f0c5d926e5\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403},{\"index\":\"security_event\",\"type\":\"_doc\",\"id\":\"280ff8cc-f43a-40ee-9ae3-9af48960832b\",\"cause\":{\"type\":\"cluster_block_exception\",\"reason\":\"index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)];\"},\"status\":403}]}

排查思路。之前有一個清理數據的接口。用來清理大屏的歷史數據。運維人員根據往常一樣執行接口。但是報錯了反饋到我這里。

一開始看到報錯有很多403。聯想到之前這個環境做了安全檢測。關閉了很多端口。之前es沒有認證以為被掃到了，詢問運維人員得知es在內部網絡并沒有開放出去。于是重新審視問題報錯。看到

index [security_event] blocked by: [FORBIDDEN/12/index read-only / allow delete (api)]

索引是只讀模式不允許刪除，但是并沒有人去主動操作es。深入查詢后發現es會檢測磁盤使用率達到85%(默認)會將索引模型調整。檢查磁盤后確實使用率達到了95%

解決方案：

臨時解除只讀限制

curl -X PUT "localhost:9200/security_event/_settings" -H 'Content-Type: application/json' -d' { "index.blocks.read_only_allow_delete": null } '

永久配置磁盤水印

curl -X PUT "localhost:9200/_cluster/settings" -H 'Content-Type: application/json' -d' { "persistent": { "cluster.routing.allocation.disk.watermark.low": "85%", "cluster.routing.allocation.disk.watermark.high": "90%", "cluster.routing.allocation.disk.watermark.flood_stage": "95%" } } '

后續交給運維人員清理磁盤數據。

Kafka 多監聽/網絡隔離轉發

老大難問題