異步日志系統設計demo

目錄

  • 簡單版本1
  • 優化版本1
  • 優化版本2

對于QPS要求很高或者對性能有一定要求的服務器程序，同步寫日志會對服務的關鍵性邏輯的快速執行和及時響應帶來一定的性能損失，因為寫日志時等待磁盤IO完成工作也需要一定時間。為了減少這種損失，一般采用異步寫日志。
本質上仍然是一個生產者與消費者模型，產生日志的線程是生產者，將日志寫入文件的線程是消費者。
如果有多個消費者線程，可能存在寫日志的時間順序錯位，所以一般將日志消費者線程數量設置為1.

簡單版本1

下面給出一個簡單的版本：

#include <iostream>
#include <thread>
#include <mutex>
#include <list>
#include <string>
#include <sstream>
#include <vector>
// 保護隊列的mutex
std::mutex log_mutex;
std::list<std::string> cached_logs;
FILE* log_file = nullptr;bool init_log_file()
{// 以追加的模式寫入文件，如果文件不存在，則創建log_file = fopen("my.log","a+");return log_file != nullptr;
}void uninit_log_file()
{if (log_file != nullptr)fclose(log_file);
}bool write_log_tofile(const std::string& line)
{if (log_file == nullptr)return false;if (fwrite((void *)line.c_str(), 1, line.length(), log_file) != line.length())return false;// 將日志flush到文件中fflush(log_file);return true;
}void log_producer()
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";{std::lock_guard<std::mutex> lock(log_mutex);cached_logs.emplace_back(os.str());}// 生產出一個log之后，休眠100ms再生產std::chrono::milliseconds duration(100);std::this_thread::sleep_for(duration);}
}void log_consumer()
{std::string line;while (true) {{std::lock_guard<std::mutex> lock(log_mutex);if (!cached_logs.empty()) {line = cached_logs.front();cached_logs.pop_front();}}// 如果取出來的行為空，說明隊列里面是空的，消費者休眠一會兒再去消費if (line.empty()) {std::chrono::milliseconds duration(1000);std::this_thread::sleep_for(duration);continue;}// 否則將line寫入到log_file中write_log_tofile(line);line.clear();}
}int main(int argc, char* argv[])
{if (!init_log_file()) {std::cout << "init log file error." << std::endl;return -1;}std::thread log_producer1(log_producer);std::thread log_producer2(log_producer);std::thread log_producer3(log_producer);std::thread log_consumer1(log_consumer);std::thread log_consumer2(log_consumer);std::thread log_consumer3(log_consumer);log_producer1.join();log_producer2.join();log_producer3.join();log_consumer1.join();log_consumer2.join();log_consumer3.join();uninit_log_file();return 0;
}

效果：

This is log, index :1, producer threadID:139910877185792
This is log, index :1, producer threadID:139910868793088
This is log, index :1, producer threadID:139910860400384
This is log, index :2, producer threadID:139910877185792
This is log, index :2, producer threadID:139910860400384
This is log, index :3, producer threadID:139910877185792
This is log, index :3, producer threadID:139910860400384
This is log, index :2, producer threadID:139910868793088
This is log, index :3, producer threadID:139910868793088
This is log, index :4, producer threadID:139910877185792
This is log, index :4, producer threadID:139910860400384
This is log, index :4, producer threadID:139910868793088
This is log, index :5, producer threadID:139910877185792
This is log, index :5, producer threadID:139910860400384
This is log, index :5, producer threadID:139910868793088
This is log, index :6, producer threadID:139910860400384
This is log, index :6, producer threadID:139910868793088
This is log, index :7, producer threadID:139910877185792
This is log, index :7, producer threadID:139910860400384
This is log, index :7, producer threadID:139910868793088
This is log, index :8, producer threadID:139910877185792
This is log, index :8, producer threadID:139910860400384
This is log, index :8, producer threadID:139910868793088
This is log, index :9, producer threadID:139910877185792
This is log, index :9, producer threadID:139910860400384
This is log, index :9, producer threadID:139910868793088
This is log, index :6, producer threadID:139910877185792
This is log, index :10, producer threadID:139910860400384
This is log, index :10, producer threadID:139910868793088
This is log, index :10, producer threadID:139910877185792
This is log, index :11, producer threadID:139910860400384
This is log, index :11, producer threadID:139910868793088
This is log, index :12, producer threadID:139910860400384
This is log, index :12, producer threadID:139910868793088
This is log, index :11, producer threadID:139910877185792
This is log, index :12, producer threadID:139910877185792
This is log, index :13, producer threadID:139910860400384
This is log, index :13, producer threadID:139910868793088
This is log, index :13, producer threadID:139910877185792
This is log, index :14, producer threadID:139910868793088
This is log, index :14, producer threadID:139910877185792
This is log, index :15, producer threadID:139910868793088
This is log, index :15, producer threadID:139910877185792
This is log, index :14, producer threadID:139910860400384
This is log, index :15, producer threadID:139910860400384
This is log, index :16, producer threadID:139910877185792
This is log, index :16, producer threadID:139910860400384
This is log, index :17, producer threadID:139910877185792
This is log, index :17, producer threadID:139910860400384
This is log, index :17, producer threadID:139910868793088
This is log, index :16, producer threadID:139910868793088
This is log, index :18, producer threadID:139910877185792
This is log, index :19, producer threadID:139910860400384
This is log, index :19, producer threadID:139910877185792
This is log, index :19, producer threadID:139910868793088
This is log, index :20, producer threadID:139910860400384
This is log, index :18, producer threadID:139910860400384
This is log, index :20, producer threadID:139910877185792
This is log, index :18, producer threadID:139910868793088
This is log, index :20, producer threadID:139910868793088
This is log, index :21, producer threadID:139910868793088
This is log, index :21, producer threadID:139910877185792
This is log, index :21, producer threadID:139910860400384
This is log, index :22, producer threadID:139910860400384
This is log, index :22, producer threadID:139910877185792
This is log, index :22, producer threadID:139910868793088
This is log, index :23, producer threadID:139910860400384
This is log, index :23, producer threadID:139910877185792
This is log, index :23, producer threadID:139910868793088
This is log, index :24, producer threadID:139910860400384
This is log, index :24, producer threadID:139910868793088
This is log, index :24, producer threadID:139910877185792
This is log, index :25, producer threadID:139910860400384
This is log, index :25, producer threadID:139910868793088
This is log, index :25, producer threadID:139910877185792
This is log, index :26, producer threadID:139910868793088
This is log, index :26, producer threadID:139910860400384
This is log, index :26, producer threadID:139910877185792
This is log, index :27, producer threadID:139910868793088
This is log, index :27, producer threadID:139910860400384
This is log, index :27, producer threadID:139910877185792
This is log, index :28, producer threadID:139910868793088
This is log, index :28, producer threadID:139910877185792
This is log, index :28, producer threadID:139910860400384
This is log, index :29, producer threadID:139910877185792
This is log, index :29, producer threadID:139910868793088
This is log, index :29, producer threadID:139910860400384
This is log, index :30, producer threadID:139910877185792
This is log, index :30, producer threadID:139910868793088
This is log, index :30, producer threadID:139910860400384
This is log, index :31, producer threadID:139910868793088
This is log, index :31, producer threadID:139910877185792
This is log, index :31, producer threadID:139910860400384
This is log, index :32, producer threadID:139910877185792
This is log, index :32, producer threadID:139910868793088
This is log, index :32, producer threadID:139910860400384
This is log, index :33, producer threadID:139910860400384
This is log, index :33, producer threadID:139910868793088
This is log, index :33, producer threadID:139910877185792
This is log, index :34, producer threadID:139910860400384
This is log, index :34, producer threadID:139910877185792
This is log, index :34, producer threadID:139910868793088
This is log, index :35, producer threadID:139910860400384
This is log, index :35, producer threadID:139910868793088
This is log, index :35, producer threadID:139910877185792
This is log, index :36, producer threadID:139910877185792
This is log, index :36, producer threadID:139910868793088
This is log, index :37, producer threadID:139910860400384
This is log, index :37, producer threadID:139910877185792
This is log, index :37, producer threadID:139910868793088
This is log, index :38, producer threadID:139910860400384
This is log, index :38, producer threadID:139910877185792
This is log, index :38, producer threadID:139910868793088
This is log, index :39, producer threadID:139910860400384
This is log, index :39, producer threadID:139910877185792
This is log, index :39, producer threadID:139910868793088
This is log, index :40, producer threadID:139910860400384
This is log, index :40, producer threadID:139910877185792
This is log, index :40, producer threadID:139910868793088
This is log, index :36, producer threadID:139910860400384
This is log, index :41, producer threadID:139910860400384
This is log, index :41, producer threadID:139910877185792
This is log, index :42, producer threadID:139910860400384
This is log, index :42, producer threadID:139910877185792
This is log, index :42, producer threadID:139910868793088
This is log, index :43, producer threadID:139910860400384
This is log, index :43, producer threadID:139910868793088
This is log, index :43, producer threadID:139910877185792
This is log, index :44, producer threadID:139910860400384
This is log, index :44, producer threadID:139910868793088
This is log, index :44, producer threadID:139910877185792
This is log, index :45, producer threadID:139910860400384
This is log, index :45, producer threadID:139910868793088
This is log, index :45, producer threadID:139910877185792
This is log, index :46, producer threadID:139910860400384
This is log, index :46, producer threadID:139910868793088
This is log, index :46, producer threadID:139910877185792
This is log, index :47, producer threadID:139910860400384
This is log, index :47, producer threadID:139910868793088
This is log, index :47, producer threadID:139910877185792
This is log, index :48, producer threadID:139910860400384
This is log, index :48, producer threadID:139910868793088
This is log, index :48, producer threadID:139910877185792
This is log, index :49, producer threadID:139910860400384
This is log, index :49, producer threadID:139910877185792
This is log, index :49, producer threadID:139910868793088
This is log, index :50, producer threadID:139910877185792
This is log, index :50, producer threadID:139910860400384
This is log, index :50, producer threadID:139910868793088
This is log, index :41, producer threadID:139910868793088

優化版本1

上面的代碼，在當前緩存隊列沒有日志記錄的時候，消費日志線程會做無用功。
這里可以使用條件變量，如果當前隊列中沒有日志記錄，就將日志消費者線程掛起；
當產生了新的日志后，signal條件變量，喚醒消費線程，將被日志從隊列中取出，并寫入文件。
下面是主要修改

#include <condition_variable>
std::condition_variable log_cv;
void log_producer()
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";{std::lock_guard<std::mutex> lock(log_mutex);cached_logs.emplace_back(os.str());log_cv.notify_one();}// 生產出一個log之后，休眠100ms再生產std::chrono::milliseconds duration(100);std::this_thread::sleep_for(duration);}
}void log_consumer()
{std::string line;while (true) {{std::unique_lock<std::mutex> lock(log_mutex);while (cached_logs.empty()) {// 無限等待log_cv.wait(lock);}line = cached_logs.front();cached_logs.pop_front();}// 如果取出來的行為空，說明隊列里面是空的，消費者休眠一會兒再去消費if (line.empty()) {std::chrono::milliseconds duration(1000);std::this_thread::sleep_for(duration);continue;}// 否則將line寫入到log_file中write_log_tofile(line);line.clear();}
}

優化版本2

還可以使用信號量來設計異步日志系統。
信號量是帶有資源計數的線程同步對象，每產生一條日志，就將信號量資源計數+1，日志消費線程默認等待這個信號量是否signal，如果signal，就喚醒一個日志消費線程，信號量計數自動-1。如果當前資源計數為0，則將消費者自動掛起。
C++現在還沒有提供不同平臺的信號量對象封裝，這里以Linux系統為例：
明顯能感覺運行相同時間，寫入的日志更多了。。

#include <unistd.h>
#include <iostream>
#include <thread>
#include <mutex>
#include <list>
#include <string>
#include <sstream>
#include <semaphore.h>// 保護隊列的mutex
pthread_mutex_t log_mutex = PTHREAD_MUTEX_INITIALIZER;
sem_t log_semphore;
std::list<std::string> cached_logs;
FILE* log_file = nullptr;bool init()
{pthread_mutex_init(&log_mutex, nullptr);// 初始信號量資源數量為0sem_init(&log_semphore, 0, 0);// 以追加的模式寫入文件，如果文件不存在，則創建log_file = fopen("my.log","a++");return log_file != nullptr;
}void uninit()
{pthread_mutex_destroy(&log_mutex);sem_destroy(&log_semphore);if (log_file != nullptr)fclose(log_file);
}bool write_log_tofile(const std::string& line)
{if (log_file == nullptr)return false;// 對于比較長的日志應該分段寫入，因為單次寫入可能只寫入部分內容// 這里邏輯從簡if (fwrite((void *)line.c_str(), 1, line.length(), log_file) != line.length())return false;// 將日志flush到文件中fflush(log_file);return true;
}void* log_producer(void* arg)
{int index = 0;while (true) {++index;std::ostringstream os;os << "This is log, index :" << index << ", producer threadID:" << std::this_thread::get_id() << "\n";pthread_mutex_lock(&log_mutex);cached_logs.emplace_back(os.str());pthread_mutex_unlock(&log_mutex);sem_post(&log_semphore);usleep(1000);}
}void* log_consumer(void* arg)
{std::string line;while (true) {// 無限等待sem_wait(&log_semphore);pthread_mutex_lock(&log_mutex);if (!cached_logs.empty()) {line = cached_logs.front();cached_logs.pop_front();}pthread_mutex_unlock(&log_mutex);// 如果取出來的行為空，說明隊列里面是空的，消費者休眠一會兒再去消費if (line.empty()) {sleep(1);continue;}// 否則將line寫入到log_file中write_log_tofile(line);line.clear();}
}int main(int argc, char* argv[])
{if (!init()) {std::cout << "init log file error." << std::endl;return -1;}// 創建三個生產日志線程pthread_t producer_thread_id[3];for (size_t i = 0; i < sizeof(producer_thread_id) / sizeof(producer_thread_id[0]); ++i) {pthread_create(&producer_thread_id[i], NULL, log_producer, NULL);}// 創建三個消費日志線程pthread_t consumer_thread_id[3];for (size_t i = 0; i < sizeof(consumer_thread_id) / sizeof(consumer_thread_id[0]); ++i) {pthread_create(&consumer_thread_id[i], NULL, log_consumer, NULL);}// 等待生產者線程退出for (size_t i = 0; i < sizeof(producer_thread_id) / sizeof(producer_thread_id[0]); ++i) {pthread_join(producer_thread_id[i], NULL);}// 等待消費者線程退出for (size_t i = 0; i < sizeof(consumer_thread_id) / sizeof(consumer_thread_id[0]); ++i) {pthread_join(consumer_thread_id[i], NULL);}uninit();return 0;
}

本文來自互聯網用戶投稿，該文觀點僅代表作者本人，不代表本站立場。本站僅提供信息存儲空間服務，不擁有所有權，不承擔相關法律責任。
如若轉載，請注明出處：http://www.pswp.cn/news/376788.shtml
繁體地址，請注明出處：http://hk.pswp.cn/news/376788.shtml
英文地址，請注明出處：http://en.pswp.cn/news/376788.shtml

如若內容造成侵權/違法違規/事實不符，請聯系多彩編程網進行投訴反饋email:809451989@qq.com，一經查實，立即刪除！

相關文章

js unix時間戳轉換

js unix時間戳轉換

一、unix時間戳轉普通時間&#xff1a; var unixtime1358932051; var unixTimestamp new Date(unixtime* 1000); commonTime unixTimestamp.toLocaleString(); alert("普通時間為&#xff1a;"commonTime); 二、普通時間轉unix時間戳 var str "2013-01-01 00…
閱讀更多...
hdu 1025（最長非遞減子序列的n*log(n)求法）

hdu 1025（最長非遞減子序列的n*log(n)求法）

題目鏈接&#xff1a;http://acm.hdu.edu.cn/showproblem.php?pid1025 經典題。。。最長非遞減序列的n*log(n)求法。。。orz... View Code 1 #include<iostream>2 const int N500007;3 using namespace std;4 int city[N];5 int dp[N];//dp[i]保存的是長度為i的最長不降…
閱讀更多...
消息隊列重要機制講解以及MQ設計思路（kafka、rabbitmq、rocketmq）

消息隊列重要機制講解以及MQ設計思路（kafka、rabbitmq、rocketmq）

目錄《Kafka篇》簡述kafka的架構設計原理&#xff08;入口點&#xff09;消息隊列有哪些作用&#xff08;簡單&#xff09;消息隊列的優缺點&#xff0c;使用場景&#xff08;基礎&#xff09;消息隊列如何保證消息可靠傳輸死信隊列是什么&#xff1f;延時隊列是什么&#xff1…
閱讀更多...
js判斷手機瀏覽器

js判斷手機瀏覽器

最新瀏覽器識別合并。 demo&#xff1a;http://v.qq.com -> http://v.qq.com/h5　　　　http://v.qq.com/ -> http://v.qq.com/h5　　　　http://v.qq.com/h5 -> http://v.qq.com/h5 <script type"text/javascript"> (function(W){ …
閱讀更多...
數據庫歸檔模式

數據庫歸檔模式

1、在sys身份下登陸oracle&#xff0c;執行命令archive log list; SQL> archive log list; Database log mode Archive Mode Automatic archival Enabled Archive destination USE_DB_RECOVERY_FILE_DEST Oldest online log sequence …
閱讀更多...
轉載|網絡編程中阻塞式函數的底層邏輯

轉載|網絡編程中阻塞式函數的底層邏輯

逛知乎看到的&#xff0c;覺得寫的挺透徹的&#xff0c;轉載一下&#xff0c;原文鏈接&#xff1a;Unix網絡編程里的阻塞是在操作系統的內核態創建一個線程來死循環嗎&#xff1f; 原文以阻塞式的recv函數作為講解&#xff0c;但是所有阻塞式的api底層邏輯基本相通。 下面是正文…
閱讀更多...
把txt文件中的json字符串寫到plist文件中

把txt文件中的json字符串寫到plist文件中

- (void)json2Plist {NSString *filePath [self applicationDocumentsDirectoryFileName:"json"];NSMutableArray *tempArray [[NSMutableArray alloc] initWithContentsOfFile:filePath];//第一次添加數據時,數組為空if (tempArray.count 0) {tempArray [NSMuta…
閱讀更多...
樹的存儲結構2 - 數據結構和算法42

樹的存儲結構2 - 數據結構和算法42

樹的存儲結構 讓編程改變世界 Change the world by program 孩子表示法 我們這次換個角度來考慮&#xff0c;由于樹中每個結點可能有多棵子樹&#xff0c;可以考慮用多重鏈表來實現。 就像我們雖然有計劃生育&#xff0c;但我們還是無法確保每個家庭只養育一個孩子的沖動&a…
閱讀更多...
海量數據去重

海量數據去重

海量數據去重 一個文件中有40億條數據&#xff0c;每條數據是一個32位的數字串&#xff0c;設計算法對其去重&#xff0c;相同的數字串僅保留一個&#xff0c;內存限制1G. 方法一&#xff1a;排序 對所有數字串進行排序&#xff0c;重復的數據傳必然相鄰&#xff0c;保留第一…
閱讀更多...
Sharepoint 2013 發布功能（Publishing features）

Sharepoint 2013 發布功能（Publishing features）

一、默認情況下&#xff0c;在創建網站集時&#xff0c;只有選擇的模板為‘ Publishing Portal&#xff08;發布門戶&#xff09;’與‘ Enterprise Wiki&#xff08;企業 Wiki&#xff09;’時才默認啟用發布功能&#xff0c;如下圖所示&#xff1a; 二、發布功能包含兩塊&…
閱讀更多...
【原】android啟動時白屏或者黑屏的問題

【原】android啟動時白屏或者黑屏的問題

解決應用啟動時白屏或者黑屏的問題 由于Activity只能到onResume時&#xff0c;才能展示到前臺&#xff0c;所以&#xff0c;如果為MAIN activity設置背景的話&#xff0c;無論onCreate-onResume速度多快&#xff0c;都會出現短暫的白屏或者黑屏 其實解決的辦法很簡單&#xff0…
閱讀更多...
【草稿】windows + vscode 遠程開發

【草稿】windows + vscode 遠程開發

主要分為三個步驟&#xff1a; 1、開啟openssh服務 2、通過ssh命令連接到遠程服務器 3、通過vscode連接遠程服務器進行開發調試 ssh概念 SSH是較可靠&#xff0c;專為遠程登陸會話和其他網絡服務提供安全性得協議&#xff0c;利用ssh協議可以有效防止遠程管理過程中得信息…
閱讀更多...
POJ3185(簡單BFS，主要做測試使用)

POJ3185(簡單BFS，主要做測試使用)

沒事做水了一道POJ的簡單BFS的題目 這道題的數據范圍是20,所以狀態總數就是&#xff08;1<<20&#xff09; 第一次提交使用STL的queue&#xff0c;并且是在隊首判斷是否達到終點&#xff0c;達到終點就退出&#xff0c;超時&#xff1a;&#xff08;其實這里我是很不明白…
閱讀更多...
tomcat站點配置

tomcat站點配置

tomcat版本&#xff1a;tomcat5.5.91、打開tomcat\conf\server.xml&#xff0c;在里面找到<Engine name"Catalina" defaultHost"localhost">.....</Engine>2、在<Engine name"Catalina" defaultHost"localhost"><…
閱讀更多...
新的視頻會議模式：StarlineProject

新的視頻會議模式：StarlineProject

目錄效果展示部分用戶參與度部分技術細節機械裝置以及硬件配置。視頻系統照明人臉跟蹤壓縮和傳輸圖像渲染音頻系統step1&#xff1a;捕獲音頻step2&#xff1a;音頻去噪處理step3&#xff1a;壓縮、傳輸、解壓step4&#xff1a;渲染可以改進的點效果展示部分 〔映維網〕谷歌光場…
閱讀更多...
HDU 3934

HDU 3934

/*這是用的有旋轉卡殼的思想。 首先確定i&#xff0c;j&#xff0c;對k進行循環&#xff0c;知道找到第一個k使得cross(i,j,k)>cross(i,j,k1),如果ki進入下一次循環。 對j&#xff0c;k進行旋轉&#xff0c;每次循環之前更新最大值&#xff0c;然后固定一個j&#xff0c;同樣…
閱讀更多...
[ios] UILocalNotification實現本地的鬧鐘提醒【轉】

[ios] UILocalNotification實現本地的鬧鐘提醒【轉】

http://www.cnblogs.com/jiangshiyong/archive/2012/06/06/2538204.html轉載于:https://www.cnblogs.com/jinjiantong/archive/2013/04/01/2992624.html
閱讀更多...
sql server根據表中數據生成insert語句

sql server根據表中數據生成insert語句

幾個收藏的根據數據庫生成Insert語句的存儲過程[修正版]----根據表中數據生成insert語句的存儲過程--建立存儲過程&#xff0c;執行spGenInsertSQL 表名--感謝playyuer----感謝szyicol--CREATEproc[dbo].[spGenInsertSQL](tablenamevarchar(256))asbegindeclaresqlvarchar(8000…
閱讀更多...
Javascript eval()函數 基礎回顧

Javascript eval()函數 基礎回顧

如果您想詳細了解ev al和JSON請參考以下鏈接&#xff1a; eval &#xff1a;https://developer.mozilla.org/En/Core_JavaScript_1.5_Reference/Global_Functions/Eval JSON&#xff1a;http://www.json.org/ eval函數的工作原理 eval函數會評估一個給定的含有JavaScript代碼的…
閱讀更多...
雜感無題|

雜感無題|

今天中午和組里面的人吃飯&#xff0c;聊起了科興跳樓的事情。這事其實前幾天我華為的mentor就轉給我了&#xff0c;當時也沒太在意&#xff0c;在脈脈上看了看&#xff0c;也不知曉是誰&#xff0c;想著可能又是抑郁癥吧。 飯后依舊繞著食堂散步&#xff0c;ly說那個人好像還是…
閱讀更多...
最新文章

Copyright @ 2022~2023