九哥聊Kestrel網絡編程第二章：開發一個Fiddler

1 文章目的

本文講解基于 kestrel 開發類似 Fiddler 應用的過程，讓讀者了解 kestrel 網絡編程里面的 kestrel 中間件和 http 應用中間件。由于最終目的不是輸出完整功能的產品，所以這里只實現 Fiddler 最核心的 http 請求和響應內容查看的功能。本文章是KestrelApp 項目^[2]里面的一個 demo 的講解，希望對您有用。

2 開發順序

代理協議 kestrel 中間件
tls 協議偵測 kestrel 中間件
隧道和 http 協議偵測 kestrel 中間件
請求響應分析 http 中間件
反向代理 http 中間件
編排中間件創建服務器和應用

3 傳輸層與 kestrel 中間件

所謂傳輸層，其目的是為了讓應用協議數據安全、可靠、快速等傳輸而存在的一種協議，其特征是把應用協議的報文做為自己的負載，常見的 tcp、udp、quic、tls 等都可以理解為傳輸層協議。比如 http 協議，常見有如下的傳輸方式：

http over tcp
http over tls over tcp
http over quic over udp

3.1 Fiddler 的傳輸層

Fiddler 要處理以下三種 http 傳輸情況：

http over tcp：直接 http 請求首頁
http over proxy over tcp：代理 http 流量
http over tls over proxy over tcp：代理 https 流量

3.2 Kestrel 的中間件

kestrel 目前的傳輸層基于 tcp 或 quic 兩種，同時內置了 tls 中間件，需要調用ListenOptions.UseHttps()來使用 tls 中間件。kestrel 的中間件的表現形式為：Func<ConnectionDelegate, ConnectionDelegate>，為了使用讀者能夠簡單理解中間件，我在KestrelFramework里定義了 kestrel 中間件的變種接口，大家基于此接口來實現更多的中間件就方便很多：

///?<summary>
///?Kestrel的中間件接口
///?</summary>
public?interface?IKestrelMiddleware
{///?<summary>///?執行///?</summary>///?<param?name="next"></param>///?<param?name="context"></param>///?<returns></returns>Task?InvokeAsync(ConnectionDelegate?next,?ConnectionContext?context);
}

4 代理協議 kestrel 中間件

Filddler 最基礎的功能是它是一個 http 代理服務器，我們需要為 kestrel 編寫代理中間件，用于處理代理傳輸層。http 代理協議分兩種：普通的 http 代理和 Connect 隧道代理。兩種的報文者是遵循 http1.0 或 1.1 的文本格式，我們可以使用 kestrel 自帶的HttpParser<>來解析這些復雜的 http 文本協議。

4.1 代理特征

在中間件編程模式中，Feature是一個很重要的中間件溝通橋梁，它往往是某個中間件工作之后，留下的財產，讓之后的中間件來獲取并受益。我們的代理中間件，也設計了 IProxyFeature，告訴之后的中間件一些代理特征。

///?<summary>
///?代理Feature
///?</summary>
public?interface?IProxyFeature
{///?<summary>///?代理主機///?</summary>HostString?ProxyHost?{?get;?}///?<summary>///?代理協議///?</summary>ProxyProtocol?ProxyProtocol?{?get;?}
}///?<summary>
///?代理協議
///?</summary>
public?enum?ProxyProtocol
{///?<summary>///?無代理///?</summary>None,///?<summary>///?http代理///?</summary>HttpProxy,///?<summary>///?隧道代理///?</summary>TunnelProxy
}

4.2 代理中間件的實現

///?<summary>
///?代理中間件
///?</summary>
sealed?class?KestrelProxyMiddleware?:?IKestrelMiddleware
{private?static?readonly?HttpParser<HttpRequestHandler>?httpParser?=?new();private?static?readonly?byte[]?http200?=?Encoding.ASCII.GetBytes("HTTP/1.1?200?Connection?Established\r\n\r\n");private?static?readonly?byte[]?http400?=?Encoding.ASCII.GetBytes("HTTP/1.1?400?Bad?Request\r\n\r\n");///?<summary>///?解析代理///?</summary>///?<param?name="next"></param>///?<param?name="context"></param>///?<returns></returns>public?async?Task?InvokeAsync(ConnectionDelegate?next,?ConnectionContext?context){var?input?=?context.Transport.Input;var?output?=?context.Transport.Output;var?request?=?new?HttpRequestHandler();while?(context.ConnectionClosed.IsCancellationRequested?==?false){var?result?=?await?input.ReadAsync();if?(result.IsCanceled){break;}try{if?(ParseRequest(result,?request,?out?var?consumed)){if?(request.ProxyProtocol?==?ProxyProtocol.TunnelProxy){input.AdvanceTo(consumed);await?output.WriteAsync(http200);}else{input.AdvanceTo(result.Buffer.Start);}context.Features.Set<IProxyFeature>(request);await?next(context);break;}else{input.AdvanceTo(result.Buffer.Start,?result.Buffer.End);}if?(result.IsCompleted){break;}}catch?(Exception){await?output.WriteAsync(http400);break;}}}///?<summary>///?解析http請求///?</summary>///?<param?name="result"></param>///?<param?name="request"></param>///?<param?name="consumed"></param>///?<returns></returns>private?static?bool?ParseRequest(ReadResult?result,?HttpRequestHandler?request,?out?SequencePosition?consumed){var?reader?=?new?SequenceReader<byte>(result.Buffer);if?(httpParser.ParseRequestLine(request,?ref?reader)?&&httpParser.ParseHeaders(request,?ref?reader)){consumed?=?reader.Position;return?true;}else{consumed?=?default;return?false;}}///?<summary>///?代理請求處理器///?</summary>private?class?HttpRequestHandler?:?IHttpRequestLineHandler,?IHttpHeadersHandler,?IProxyFeature{private?HttpMethod?method;public?HostString?ProxyHost?{?get;?private?set;?}public?ProxyProtocol?ProxyProtocol{get{if?(ProxyHost.HasValue?==?false){return?ProxyProtocol.None;}if?(method?==?HttpMethod.Connect){return?ProxyProtocol.TunnelProxy;}return?ProxyProtocol.HttpProxy;}}void?IHttpRequestLineHandler.OnStartLine(HttpVersionAndMethod?versionAndMethod,?TargetOffsetPathLength?targetPath,?Span<byte>?startLine){method?=?versionAndMethod.Method;var?host?=?Encoding.ASCII.GetString(startLine.Slice(targetPath.Offset,?targetPath.Length));if?(versionAndMethod.Method?==?HttpMethod.Connect){ProxyHost?=?HostString.FromUriComponent(host);}else?if?(Uri.TryCreate(host,?UriKind.Absolute,?out?var?uri)){ProxyHost?=?HostString.FromUriComponent(uri);}}void?IHttpHeadersHandler.OnHeader(ReadOnlySpan<byte>?name,?ReadOnlySpan<byte>?value){}void?IHttpHeadersHandler.OnHeadersComplete(bool?endStream){}void?IHttpHeadersHandler.OnStaticIndexedHeader(int?index){}void?IHttpHeadersHandler.OnStaticIndexedHeader(int?index,?ReadOnlySpan<byte>?value){}}
}

5 tls 協議偵測 kestrel 中間件

Fiddler 只監聽了一個端口，要同時支持非加密和加密兩種流量，如果不調用調用ListenOptions.UseHttps()，我們的程序就不支持 https 的分析；如果直接調用ListenOptions.UseHttps()，會讓我們的程序不支持非加密的 http 的分析，這就要求我們有條件的根據客戶端發來的流量分析是否需要開啟。

我已經在KestrelFramework內置了TlsDetection中間件，這個中間件可以根據客戶端的實際流量類型來選擇是否使用 tls。在 Fiddler 中，我們還需要根據客戶端的tls握手中的sni使用 ca 證書來動態生成服務器證書用于 tls 加密傳輸。

///?<summary>
///?證書服務
///?</summary>
sealed?class?CertService
{private?const?string?CACERT_PATH?=?"cacert";private?readonly?IMemoryCache?serverCertCache;private?readonly?IEnumerable<ICaCertInstaller>?certInstallers;private?readonly?ILogger<CertService>?logger;private?X509Certificate2??caCert;///?<summary>///?獲取證書文件路徑///?</summary>public?string?CaCerFilePath?{?get;?}?=?OperatingSystem.IsLinux()???$"{CACERT_PATH}/fiddler.crt"?:?$"{CACERT_PATH}/fiddler.cer";///?<summary>///?獲取私鑰文件路徑///?</summary>public?string?CaKeyFilePath?{?get;?}?=?$"{CACERT_PATH}/fiddler.key";///?<summary>///?證書服務///?</summary>///?<param?name="serverCertCache"></param>///?<param?name="certInstallers"></param>///?<param?name="logger"></param>public?CertService(IMemoryCache?serverCertCache,IEnumerable<ICaCertInstaller>?certInstallers,ILogger<CertService>?logger){this.serverCertCache?=?serverCertCache;this.certInstallers?=?certInstallers;this.logger?=?logger;Directory.CreateDirectory(CACERT_PATH);}///?<summary>///?生成CA證書///?</summary>public?bool?CreateCaCertIfNotExists(){if?(File.Exists(this.CaCerFilePath)?&&?File.Exists(this.CaKeyFilePath)){return?false;}File.Delete(this.CaCerFilePath);File.Delete(this.CaKeyFilePath);var?notBefore?=?DateTimeOffset.Now.AddDays(-1);var?notAfter?=?DateTimeOffset.Now.AddYears(10);var?subjectName?=?new?X500DistinguishedName($"CN={nameof(Fiddler)}");this.caCert?=?CertGenerator.CreateCACertificate(subjectName,?notBefore,?notAfter);var?privateKeyPem?=?this.caCert.GetRSAPrivateKey()?.ExportRSAPrivateKeyPem();File.WriteAllText(this.CaKeyFilePath,?new?string(privateKeyPem),?Encoding.ASCII);var?certPem?=?this.caCert.ExportCertificatePem();File.WriteAllText(this.CaCerFilePath,?new?string(certPem),?Encoding.ASCII);return?true;}///?<summary>///?安裝和信任CA證書///?</summary>public?void?InstallAndTrustCaCert(){var?installer?=?this.certInstallers.FirstOrDefault(item?=>?item.IsSupported());if?(installer?!=?null){installer.Install(this.CaCerFilePath);}else{this.logger.LogWarning($"請根據你的系統平臺手動安裝和信任CA證書{this.CaCerFilePath}");}}///?<summary>///?獲取頒發給指定域名的證書///?</summary>///?<param?name="domain"></param>///?<returns></returns>public?X509Certificate2?GetOrCreateServerCert(string??domain){if?(this.caCert?==?null){using?var?rsa?=?RSA.Create();rsa.ImportFromPem(File.ReadAllText(this.CaKeyFilePath));this.caCert?=?new?X509Certificate2(this.CaCerFilePath).CopyWithPrivateKey(rsa);}var?key?=?$"{nameof(CertService)}:{domain}";var?endCert?=?this.serverCertCache.GetOrCreate(key,?GetOrCreateCert);return?endCert!;//?生成域名的1年證書X509Certificate2?GetOrCreateCert(ICacheEntry?entry){var?notBefore?=?DateTimeOffset.Now.AddDays(-1);var?notAfter?=?DateTimeOffset.Now.AddYears(1);entry.SetAbsoluteExpiration(notAfter);var?extraDomains?=?GetExtraDomains();var?subjectName?=?new?X500DistinguishedName($"CN={domain}");var?endCert?=?CertGenerator.CreateEndCertificate(this.caCert,?subjectName,?extraDomains,?notBefore,?notAfter);//?重新初始化證書，以兼容win平臺不能使用內存證書return?new?X509Certificate2(endCert.Export(X509ContentType.Pfx));}}///?<summary>///?獲取域名///?</summary>///?<param?name="domain"></param>///?<returns></returns>private?static?IEnumerable<string>?GetExtraDomains(){yield?return?Environment.MachineName;yield?return?IPAddress.Loopback.ToString();yield?return?IPAddress.IPv6Loopback.ToString();}
}

6 隧道和 http 協議偵測 kestrel 中間件

經過KestrelProxyMiddleware后的流量，在 tls 解密(如果可能)之后，一般情況下都是 http 流量了，但如果你在 qq 設置代理到我們這個偽 Fildder 之后，會發現部分流量流量不是 http 流量，原因是 http 隧道也是一個通用傳輸層，可以傳輸任意 tcp 或 tcp 之上的流量。所以我們需要新的中間件來檢測當前流量，如果不是 http 流量就回退到隧道代理的流程，即我們不跟蹤不分析這部分非 http 流量。

6.1 http 流量偵測

///?<summary>
///?流量偵測器
///?</summary>
private?static?class?FlowDetector
{private?static?readonly?byte[]?crlf?=?Encoding.ASCII.GetBytes("\r\n");private?static?readonly?byte[]?http10?=?Encoding.ASCII.GetBytes("?HTTP/1.0");private?static?readonly?byte[]?http11?=?Encoding.ASCII.GetBytes("?HTTP/1.1");private?static?readonly?byte[]?http20?=?Encoding.ASCII.GetBytes("?HTTP/2.0");///?<summary>///?傳輸內容是否為http///?</summary>///?<param?name="context"></param>///?<returns></returns>public?static?async?ValueTask<bool>?IsHttpAsync(ConnectionContext?context){var?input?=?context.Transport.Input;var?result?=?await?input.ReadAtLeastAsync(1);var?isHttp?=?IsHttp(result);input.AdvanceTo(result.Buffer.Start);return?isHttp;}private?static?bool?IsHttp(ReadResult?result){var?reader?=?new?SequenceReader<byte>(result.Buffer);if?(reader.TryReadToAny(out?ReadOnlySpan<byte>?line,?crlf)){return?line.EndsWith(http11)?||?line.EndsWith(http20)?||?line.EndsWith(http10);}return?false;}
}

6.2 隧道回退中間件

///?<summary>
///?隧道傳輸中間件
///?</summary>
sealed?class?KestrelTunnelMiddleware?:?IKestrelMiddleware
{private?readonly?ILogger<KestrelTunnelMiddleware>?logger;///?<summary>///?隧道傳輸中間件///?</summary>///?<param?name="logger"></param>public?KestrelTunnelMiddleware(ILogger<KestrelTunnelMiddleware>?logger){this.logger?=?logger;}///?<summary>///?執行中間你件///?</summary>///?<param?name="next"></param>///?<param?name="context"></param>///?<returns></returns>public?async?Task?InvokeAsync(ConnectionDelegate?next,?ConnectionContext?context){var?feature?=?context.Features.Get<IProxyFeature>();if?(feature?==?null?||?feature.ProxyProtocol?==?ProxyProtocol.None){this.logger.LogInformation($"偵測到http直接請求");await?next(context);}else?if?(feature.ProxyProtocol?==?ProxyProtocol.HttpProxy){this.logger.LogInformation($"偵測到普通http代理流量");await?next(context);}else?if?(await?FlowDetector.IsHttpAsync(context)){this.logger.LogInformation($"偵測到隧道傳輸http流量");await?next(context);}else{this.logger.LogInformation($"跳過隧道傳輸非http流量{feature.ProxyHost}的攔截");await?TunnelAsync(context,?feature);}}///?<summary>///?隧道傳輸其它協議的數據///?</summary>///?<param?name="context"></param>///?<param?name="feature"></param>///?<returns></returns>private?async?ValueTask?TunnelAsync(ConnectionContext?context,?IProxyFeature?feature){var?port?=?feature.ProxyHost.Port;if?(port?==?null){return;}try{var?host?=?feature.ProxyHost.Host;using?var?socket?=?new?Socket(SocketType.Stream,?ProtocolType.Tcp);await?socket.ConnectAsync(host,?port.Value,?context.ConnectionClosed);Stream?stream?=?new?NetworkStream(socket,?ownsSocket:?false);//?如果有tls中間件，則反回來加密隧道if?(context.Features.Get<ITlsConnectionFeature>()?!=?null){var?sslStream?=?new?SslStream(stream,?leaveInnerStreamOpen:?true);await?sslStream.AuthenticateAsClientAsync(feature.ProxyHost.Host);stream?=?sslStream;}var?task1?=?stream.CopyToAsync(context.Transport.Output);var?task2?=?context.Transport.Input.CopyToAsync(stream);await?Task.WhenAny(task1,?task2);}catch?(Exception?ex){this.logger.LogError(ex,?$"連接到{feature.ProxyHost}異常");}}
}

7 請求響應分析 http 中間件

這部分屬于 asp.netcore 應用層內容，關鍵點是制作可多次讀取的 http 請求 body 流和 http 響應 body 流，因為每個分析器實例都可以會重頭讀取一次請求內容和響應內容。

7.1 http 分析器

為了方便各種分析器的獨立實現，我們定義 http 分析器的接口

///?<summary>
///?http分析器
///?支持多個實例
///?</summary>
public?interface?IHttpAnalyzer
{///?<summary>///?分析http///?</summary>///?<param?name="context"></param>///?<returns></returns>ValueTask?AnalyzeAsync(HttpContext?context);
}

這是輸到日志的 http 分析器

public?class?LoggingHttpAnalyzer?:?IHttpAnalyzer
{private?readonly?ILogger<LoggingHttpAnalyzer>?logger;public?LoggingHttpAnalyzer(ILogger<LoggingHttpAnalyzer>?logger){this.logger?=?logger;}public?async?ValueTask?AnalyzeAsync(HttpContext?context){var?builder?=?new?StringBuilder();var?writer?=?new?StringWriter(builder);writer.WriteLine("[REQUEST]");await?context.SerializeRequestAsync(writer);writer.WriteLine("[RESPONSE]");await?context.SerializeResponseAsync(writer);this.logger.LogInformation(builder.ToString());}
}

7.2 分析 http 中間件

我們把請求 body 流和響應 body 流保存到臨時文件，在所有分析器工作之后再刪除。

///?<summary>
///?http分析中間件
///?</summary>
sealed?class?HttpAnalyzeMiddleware
{private?readonly?RequestDelegate?next;private?readonly?IEnumerable<IHttpAnalyzer>?analyzers;///?<summary>///?http分析中間件///?</summary>///?<param?name="next"></param>///?<param?name="analyzers"></param>public?HttpAnalyzeMiddleware(RequestDelegate?next,IEnumerable<IHttpAnalyzer>?analyzers){this.next?=?next;this.analyzers?=?analyzers;}///?<summary>///?分析代理的http流量///?</summary>///?<param?name="context"></param>///?<returns></returns>public?async?Task?InvokeAsync(HttpContext?context){var?feature?=?context.Features.Get<IProxyFeature>();if?(feature?==?null?||?feature.ProxyProtocol?==?ProxyProtocol.None){await?next(context);return;}context.Request.EnableBuffering();var?oldBody?=?context.Response.Body;using?var?response?=?new?FileResponse();try{//?替換response的bodycontext.Response.Body?=?response.Body;//?請求下個中間件await?next(context);//?處理分析await?this.AnalyzeAsync(context);}finally{response.Body.Position?=?0L;await?response.Body.CopyToAsync(oldBody);context.Response.Body?=?oldBody;}}private?async?ValueTask?AnalyzeAsync(HttpContext?context){foreach?(var?item?in?this.analyzers){context.Request.Body.Position?=?0L;context.Response.Body.Position?=?0L;await?item.AnalyzeAsync(context);}}private?class?FileResponse?:?IDisposable{private?readonly?string?filePath?=?Path.GetTempFileName();public?Stream?Body?{?get;?}public?FileResponse(){this.Body?=?new?FileStream(filePath,?FileMode.Open,?FileAccess.ReadWrite);}public?void?Dispose(){this.Body.Dispose();File.Delete(filePath);}}
}

8 反向代理 http 中間件

我們需要把請求轉發到真實的目標服務器，這時我們的應用程序是一個 http 客戶端角色，這個過程與 nginx 的反向代理是一致的。具體的實現上，我們直接使用 yarp 庫來完成即可。

///?<summary>
///?http代理執行中間件
///?</summary>
sealed?class?HttpForwardMiddleware
{private?readonly?RequestDelegate?next;private?readonly?IHttpForwarder?httpForwarder;private?readonly?HttpMessageInvoker?httpClient?=?new(CreateSocketsHttpHandler());///?<summary>///?http代理執行中間件///?</summary>///?<param?name="next"></param>///?<param?name="httpForwarder"></param>public?HttpForwardMiddleware(RequestDelegate?next,IHttpForwarder?httpForwarder){this.next?=?next;this.httpForwarder?=?httpForwarder;}///?<summary>///?轉發http流量///?</summary>///?<param?name="context"></param>///?<returns></returns>public?async?Task?InvokeAsync(HttpContext?context){var?feature?=?context.Features.Get<IProxyFeature>();if?(feature?==?null?||?feature.ProxyProtocol?==?ProxyProtocol.None){await?next(context);}else{var?scheme?=?context.Request.Scheme;var?destinationPrefix?=?$"{scheme}://{feature.ProxyHost}";await?httpForwarder.SendAsync(context,?destinationPrefix,?httpClient,?ForwarderRequestConfig.Empty,?HttpTransformer.Empty);}}private?static?SocketsHttpHandler?CreateSocketsHttpHandler(){return?new?SocketsHttpHandler{Proxy?=?null,UseProxy?=?false,UseCookies?=?false,AllowAutoRedirect?=?false,AutomaticDecompression?=?DecompressionMethods.None,};}
}

9 編排中間件創建服務器和應用

9.1 kestrel 中間件編排

這里要特別注意順序，傳輸層套娃。

///?<summary>
///??ListenOptions擴展
///?</summary>
public?static?partial?class?ListenOptionsExtensions
{///?<summary>///?使用Fiddler的kestrel中間件///?</summary>///?<param?name="listen"></param>public?static?ListenOptions?UseFiddler(this?ListenOptions?listen){//?代理協議中間件listen.Use<KestrelProxyMiddleware>();//?tls偵測中間件listen.UseTlsDetection(tls?=>{var?certService?=?listen.ApplicationServices.GetRequiredService<CertService>();certService.CreateCaCertIfNotExists();certService.InstallAndTrustCaCert();tls.ServerCertificateSelector?=?(context,?domain)?=>?certService.GetOrCreateServerCert(domain);});//?隧道代理處理中間件listen.Use<KestrelTunnelMiddleware>();return?listen;}
}

9.2 http 中間件的編排

public?static?class?ApplicationBuilderExtensions
{///?<summary>///?使用Fiddler的http中間件///?</summary>///?<param?name="app"></param>public?static?void?UseFiddler(this?IApplicationBuilder?app){app.UseMiddleware<HttpAnalyzeMiddleware>();app.UseMiddleware<HttpForwardMiddleware>();}
}

9.3 創建應用

我們可以在傳統的 MVC 里創建偽 fiddler 的首頁、下載證書等 http 交互頁面。

public?static?void?Main(string[]?args)
{var?builder?=?WebApplication.CreateBuilder(args);builder.Services.AddFiddler().AddControllers();builder.WebHost.ConfigureKestrel((context,?kestrel)?=>{var?section?=?context.Configuration.GetSection("Kestrel");kestrel.Configure(section).Endpoint("Fiddler",?endpoint?=>?endpoint.ListenOptions.UseFiddler());});var?app?=?builder.Build();app.UseRouting();app.UseFiddler();app.MapControllerRoute(name:?"default",pattern:?"{controller=Home}/{action=Index}/{id?}");app.Run();
}