我們都知道在分布式日志當中,exceptionless客戶端是把日志寫到
Elasticsearch數據庫,就像我們把數據寫入到關系數據庫一樣;
既然是寫入,那么在短時間大數據量的情況下,寫入就會涉及到效率的問題;
首先我們看下 exceptionless源碼
public class InMemoryObjectStorage : IObjectStorage {private readonly Dictionary<string, Tuple<ObjectInfo, object>> _storage = new Dictionary<string, Tuple<ObjectInfo, object>>(StringComparer.OrdinalIgnoreCase);private readonly object _lock = new object();public InMemoryObjectStorage() : this(1000) {}public InMemoryObjectStorage(int maxObjects) {MaxObjects = maxObjects;}public long MaxObjects { get; set; }public int Count {get { return _storage.Count; }}
我們寫入到日志的方法 ?,首先會把日志寫到內存當中;但是為了防止日志過大,導致服務器的內存被過度消耗的問題 ,這邊是限制了 日志的條數的 1000;
當有新的日志進來時,如果是超過1000,那么舊的日志是會被拋棄的;
public bool SaveObject<T>(string path, T value) where T : class {if (String.IsNullOrWhiteSpace(path))throw new ArgumentNullException("path");lock (_lock) {_storage[path] = Tuple.Create(new ObjectInfo {Created = DateTime.Now,Modified = DateTime.Now,Path = path}, (object)value);if (_storage.Count > MaxObjects)_storage.Remove(_storage.OrderByDescending(kvp => kvp.Value.Item1.Created).First().Key);}return true;}
這顯然不是 我們想要的,我們需要的是完整的日志;
public SubmissionResponse PostEvents(IEnumerable<Event> events, ExceptionlessConfiguration config, IJsonSerializer serializer) {if (!config.IsValid)return new SubmissionResponse(500, message: "Invalid client configuration settings");string data = serializer.Serialize(events);string url = String.Format("{0}/events", GetServiceEndPoint(config));HttpResponseMessage response;try {HttpContent content = new StringContent(data, Encoding.UTF8, "application/json");// don't compress data smaller than 4kbif (data.Length > 1024 * 4)content = new GzipContent(content);_client.Value.AddAuthorizationHeader(config.ApiKey);response = _client.Value.PostAsync(url, content).ConfigureAwait(false).GetAwaiter().GetResult();} catch (Exception ex) {return new SubmissionResponse(500, exception: ex);}int settingsVersion;if (Int32.TryParse(GetSettingsVersionHeader(response.Headers), out settingsVersion))SettingsManager.CheckVersion(settingsVersion, config);return new SubmissionResponse((int)response.StatusCode, GetResponseMessage(response));}
所以我們需要在PostEvents 這個方法做文章;
目前我們的做法是:
就是把日志直接寫到消息隊列,然后做一個消費日志的訂閱隊列的服務,從隊列讀取數據,在寫到Elasticsearch;