概述
- 基于前文,我們知道如何集成多租戶的相關功能了, 現在我們繼續集成Monodb的多租戶形式
- 需要注意的是,MongoDB 在 NestJS 中的使用過程中存在一些“坑點”
- 如果按照默認方式集成,會發現連接數在不斷增長,即使我們請求的是相同的數據庫地址和租戶信息
- 這說明每次接口請求都在創建新的數據庫連接,而不是復用已有連接,這個行為與我們預期不符
- 數據庫連接是有限資源,頻繁創建連接會導致 IO 資源耗盡、性能下降等問題
- 下面,我們分別來看看如何解決此類問題
開始集成
1 ) 編寫 docker-compose.multi.yaml 文件
services:mongo:image: mongo:8 # 使用最新的 MongoDB 鏡像container_name: mongo_apprestart: alwaysports:- "27018:27017" # 將容器的 27017 端口映射到主機的 27017 端口environment:- MONGO_INITDB_ROOT_USERNAME=root # 設置 MongoDB 的 root 用戶名- MONGO_INITDB_ROOT_PASSWORD=123456_mongodb # 設置 MongoDB 的 root 密碼# 調整日志級別的例子,可選值如 "0"(致命錯誤)、"1"(警告+錯誤)、"2"(信息+警告+錯誤)...- MONGO_LOG_LEVEL=2- MONGO_SYSTEM_LOG_PATH=/data/logs/mongodb.logvolumes:- ./docker-dbconfig/mongo/conf/mongod.conf:/etc/mongod.conf- ./docker-dbconfig/mongo/data/db:/data/db # 將容器內的 /data/db 目錄映射到本地的 ./data/db 目錄,用于數據持久化- ./docker-dbconfig/mongo/logs:/data/logsnetworks:- light_networkmongo-express:image: mongo-express:latestcontainer_name: mongo_express_apprestart: alwaysenvironment:ME_CONFIG_MONGODB_ADMINUSERNAME: rootME_CONFIG_MONGODB_ADMINPASSWORD: 123456_mongodbME_CONFIG_MONGODB_URL: mongodb://root:123456_mongodb@mongo:27017/ME_CONFIG_BASICAUTH: falseports:- 18081:8081networks:- light_networkmongo2:image: mongo:8 # 使用最新的 MongoDB 鏡像container_name: mongo_app2restart: alwaysports:- "27019:27017" # 將容器的 27017 端口映射到主機的 27017 端口environment:- MONGO_INITDB_ROOT_USERNAME=root # 設置 MongoDB 的 root 用戶名- MONGO_INITDB_ROOT_PASSWORD=123456_mongodb # 設置 MongoDB 的 root 密碼# 調整日志級別的例子,可選值如 "0"(致命錯誤)、"1"(警告+錯誤)、"2"(信息+警告+錯誤)...- MONGO_LOG_LEVEL=2- MONGO_SYSTEM_LOG_PATH=/data/logs/mongodb.logvolumes:- ./docker-dbconfig/mongo2/conf/mongod.conf:/etc/mongod.conf- ./docker-dbconfig/mongo2/data/db:/data/db # 將容器內的 /data/db 目錄映射到本地的 ./data/db 目錄,用于數據持久化- ./docker-dbconfig/mongo2/logs:/data/logsnetworks:- light_networkmongo-express2:image: mongo-express:latestcontainer_name: mongo_express_app2restart: alwaysenvironment:ME_CONFIG_MONGODB_ADMINUSERNAME: rootME_CONFIG_MONGODB_ADMINPASSWORD: 123456_mongodbME_CONFIG_MONGODB_URL: mongodb://root:123456_mongodb@mongo2:27017/ME_CONFIG_BASICAUTH: falseports:- 18082:8081networks:- light_networknetworks:light_network:external: true
- 啟動服務,之后在 UI 管理界面給 2個數據庫加入可識別的數據
- 下面測試的時候,會看到數據
2 ) 配置 .env
T1_MONGODB_URI="mongodb://root:123456_mongodb@localhost:27018/nestmongodb"
T2_MONGODB_URI="mongodb://root:123456_mongodb@localhost:27019/nestmongodb"
3 ) 編寫 database/mongoose/mongoose.constant.ts
// 這個配置模擬調接口/讀數據庫獲取的
export const tenantMap = new Map([['1', 'T1'],['2', 'T2']
]);
export const defaultTenant = tenantMap.values().next().value; // 第一個
4 ) 編寫 database/mongoose/mongoose-config.service.ts
import { Inject } from '@nestjs/common';
import { REQUEST } from '@nestjs/core';
import {MongooseModuleOptions,MongooseOptionsFactory,
} from '@nestjs/mongoose';
import { Request } from 'express';
import { tenantMap, defaultTenant } from './mongoose.constant';
import { ConfigService } from '@nestjs/config';export class MongooseConfigService implements MongooseOptionsFactory {constructor(@Inject(REQUEST) private request: Request,private configService: ConfigService,) {}createMongooseOptions():| MongooseModuleOptions| Promise<MongooseModuleOptions> {const headers = this.request.headers;const tenantId = headers['x-tenant-id'] as string;console.log('tenantId: ', tenantId)if (tenantId && !tenantMap.has(tenantId)) {throw new Error('invalid tenantId');}const t_prefix = !tenantId ? defaultTenant : tenantMap.get(tenantId);const uri = this.configService.get<string>(`${t_prefix}_MONGODB_URI`);return { uri } as MongooseModuleOptions;}
}
5 ) 編寫 app.module.ts
import { Module } from '@nestjs/common';
import { AppController } from './app.controller';
import { MongooseModule } from '@nestjs/mongoose';
import { UserSchema } from './user/user.schema'
import { ConfigModule, ConfigService } from '@nestjs/config';
import { MongooseConfigService } from './database/mongoose/mongoose-config.service';@Module({imports: [// 1. 下面這個后續可以封裝一個新的模塊,來匹配 .env 和 其他配置ConfigModule.forRoot({ // 配置環境變量模塊envFilePath: '.env', // 指定環境變量文件路徑isGlobal: true, // 全局可用}),MongooseModule.forRootAsync({useClass: MongooseConfigService,}),MongooseModule.forFeature([{ name: 'User', schema: UserSchema }]),],controllers: [AppController],providers: []
})export class AppModule {}
6 ) 編寫 app.controller.ts
import { Controller, Get } from '@nestjs/common';
import { InjectRepository } from '@nestjs/typeorm';
import { User } from './user/user.entity';
import { InjectModel } from '@nestjs/mongoose';
import { Model, Document as Doc } from 'mongoose';@Controller()
export class AppController {constructor(@InjectModel('User') private userModel: Model<Doc>,) {}@Get('/multi-mongoose')async getMongooseUsers(): Promise<any> {const rs = await this.userModel.find()return rs;}
}
測試
1 ) 測試1
-
請求
curl --request GET \--url http://localhost:3000/multi-mongoose \--header 'x-tenant-id: 1'
-
響應
[{"_id": "6874d4b0d10e36e350dd588d","username": "mongo1","password": "123456"},{"_id": "6874d4d9d10e36e350dd588f","username": "lee","password": "123456"} ]
2 ) 測試2
-
請求
curl --request GET \--url http://localhost:3000/multi-mongoose \--header 'x-tenant-id: 2'
-
響應
[{"_id": "6874d4b0d10e36e350dd588d","username": "mongo2","password": "123456"},{"_id": "6874d4d9d10e36e350dd588f","username": "lee","password": "123456"} ]
3 )進入其中一個數據庫,測試連接情況
docker exec -it mongo_app2 mongosh admin -u root # 輸入密碼
use nestmongodb;# 其實上面一個 use 命令 可以不用
db.serverStatus().connections
輸出如下:
{current: 6,available: 838839,totalCreated: 38,rejected: 0,active: 2,threaded: 6,exhaustIsMaster: Long('0'),exhaustHello: Long('0'),awaitingTopologyChanges: Long('1'),loadBalanced: Long('0')
}
目前 active 有2個,當不斷請求同一個接口,在此執行上述命令,可發現這個數字是累加的
這明顯是不對的,訪問相同鏈接應該使用同一個 connection, 而不是創建新的通道
當租戶多起來的時候,會帶來嚴重的性能問題
4 ) 分析原因
在 @nestjs/mongoose
包的 mongoose-core.module.ts 中 在 使用 factory 方法的時候,會調用 createMongooseConnection
private static async createMongooseConnection(uri: string,mongooseOptions: ConnectOptions,factoryOptions: {lazyConnection?: boolean;onConnectionCreate?: MongooseModuleOptions['onConnectionCreate'];},
): Promise<Connection> {const connection = mongoose.createConnection(uri, mongooseOptions);if (factoryOptions?.lazyConnection) {return connection;}factoryOptions?.onConnectionCreate?.(connection);return connection.asPromise();
}
這個是每次都會創建的根本原因,這個包也沒有提供 類似 datasourceFactory 的功能
現在需要定制 mongoose 的 module 中的 forRootSync 的邏輯
定制 mongoose 的 forRootAsync 方法
現在需要對官方 @nestjs/mongoose 包中的核心模塊的方法進行裁剪和優化
找到對應的文件貼到自己項目中進行修改
1 )新建 src/database/mongoose/mongoose.module.ts
import { Module, DynamicModule } from '@nestjs/common';
import { MongooseModuleAsyncOptions, MongooseModuleOptions, MongooseModule as NestMongooseModule
} from '@nestjs/mongoose';
import { MongooseCoreModule } from './mongoose-core.module';@Module({})
export class MongooseModule extends NestMongooseModule {static forRoot(uri: string,options: MongooseModuleOptions = {},): DynamicModule {return {module: MongooseModule,imports: [MongooseCoreModule.forRoot(uri, options)],}}static forRootAsync(options: MongooseModuleAsyncOptions): DynamicModule {return {module: MongooseModule,imports: [MongooseCoreModule.forRootAsync(options)],}}
}
這是最外層,用于引入 MongooseCoreModule
模塊, 重寫有問題的源碼
2 )新建 src/database/mongoose/mongoose-core.module.ts
import {DynamicModule,Global,Inject,Module,OnApplicationShutdown,Provider,Type,
} from '@nestjs/common';
import { ModuleRef } from '@nestjs/core';
import * as mongoose from 'mongoose';
import { ConnectOptions, Connection } from 'mongoose';
import { defer, lastValueFrom } from 'rxjs';
import { catchError } from 'rxjs/operators';
import { handleRetry } from './mongoose.utils';
import {MONGOOSE_CONNECTION_NAME,MONGOOSE_MODULE_OPTIONS,
} from './mongoose.constants';import {MongooseModuleOptions,MongooseModuleAsyncOptions,MongooseModuleFactoryOptions,MongooseOptionsFactory,getConnectionToken,
} from '@nestjs/mongoose';@Global()
@Module({})
export class MongooseCoreModule implements OnApplicationShutdown {private static connections: Record<string, mongoose.Connection> = {};constructor(@Inject(MONGOOSE_CONNECTION_NAME) private readonly connectionName: string,private readonly moduleRef: ModuleRef,) {}static forRoot(uri: string,options: MongooseModuleOptions = {},): DynamicModule {const {retryAttempts,retryDelay,connectionName,connectionFactory,connectionErrorFactory,lazyConnection,onConnectionCreate,verboseRetryLog,...mongooseOptions} = options;const mongooseConnectionFactory =connectionFactory || ((connection) => connection);const mongooseConnectionError =connectionErrorFactory || ((error) => error);const mongooseConnectionName = getConnectionToken(connectionName);const mongooseConnectionNameProvider = {provide: MONGOOSE_CONNECTION_NAME,useValue: mongooseConnectionName,};const connectionProvider = {provide: mongooseConnectionName,useFactory: async (): Promise<any> =>await lastValueFrom(defer(async () =>mongooseConnectionFactory(await this.createMongooseConnection(uri, mongooseOptions, {lazyConnection,onConnectionCreate,}),mongooseConnectionName,),).pipe(handleRetry(retryAttempts, retryDelay, verboseRetryLog),catchError((error) => {throw mongooseConnectionError(error);}),),),};return {module: MongooseCoreModule,providers: [connectionProvider, mongooseConnectionNameProvider],exports: [connectionProvider],};}static forRootAsync(options: MongooseModuleAsyncOptions): DynamicModule {const mongooseConnectionName = getConnectionToken(options.connectionName);const mongooseConnectionNameProvider = {provide: MONGOOSE_CONNECTION_NAME,useValue: mongooseConnectionName,};const connectionProvider = {provide: mongooseConnectionName,useFactory: async (mongooseModuleOptions: MongooseModuleFactoryOptions,): Promise<any> => {const {retryAttempts,retryDelay,uri,connectionFactory,connectionErrorFactory,lazyConnection,onConnectionCreate,verboseRetryLog,...mongooseOptions} = mongooseModuleOptions;const mongooseConnectionFactory =connectionFactory || ((connection) => connection);const mongooseConnectionError =connectionErrorFactory || ((error) => error);return await lastValueFrom(defer(async () =>mongooseConnectionFactory(await this.createMongooseConnection(uri as string,mongooseOptions,{ lazyConnection, onConnectionCreate },),mongooseConnectionName,),).pipe(handleRetry(retryAttempts, retryDelay, verboseRetryLog),catchError((error) => {throw mongooseConnectionError(error);}),),);},inject: [MONGOOSE_MODULE_OPTIONS],};const asyncProviders = this.createAsyncProviders(options);return {module: MongooseCoreModule,imports: options.imports,providers: [...asyncProviders,connectionProvider,mongooseConnectionNameProvider,],exports: [connectionProvider],};}private static createAsyncProviders(options: MongooseModuleAsyncOptions,): Provider[] {if (options.useExisting || options.useFactory) {return [this.createAsyncOptionsProvider(options)];}const useClass = options.useClass as Type<MongooseOptionsFactory>;return [this.createAsyncOptionsProvider(options),{provide: useClass,useClass,},];}private static createAsyncOptionsProvider(options: MongooseModuleAsyncOptions,): Provider {if (options.useFactory) {return {provide: MONGOOSE_MODULE_OPTIONS,useFactory: options.useFactory,inject: options.inject || [],};}// `as Type<MongooseOptionsFactory>` is a workaround for microsoft/TypeScript#31603const inject = [(options.useClass || options.useExisting) as Type<MongooseOptionsFactory>,];return {provide: MONGOOSE_MODULE_OPTIONS,useFactory: async (optionsFactory: MongooseOptionsFactory) =>await optionsFactory.createMongooseOptions(),inject,};}private static async createMongooseConnection(uri: string,mongooseOptions: ConnectOptions,factoryOptions: {lazyConnection?: boolean;onConnectionCreate?: MongooseModuleOptions['onConnectionCreate'];},): Promise<Connection> {// 添加這里if (this.connections[uri]) {return this.connections[uri];}const connection = mongoose.createConnection(uri, mongooseOptions);this.connections[uri] = connection; // 這里賦值if (factoryOptions?.lazyConnection) {return connection;}factoryOptions?.onConnectionCreate?.(connection);return connection.asPromise();}async onApplicationShutdown() {const connection = this.moduleRef.get<any>(this.connectionName);if (connection) {await connection.close();}const connectionClients = Object.values(MongooseCoreModule.connections);if (connectionClients.length > 0) {// 銷毀所有 mongoose connectionfor (const client of connectionClients) {client?.close();}}}
}
這里,注意 createMongooseConnection
以及 onApplicationShutdown
中的 處理
對連接進行優化處理,以及在異常關閉時對客戶端進行銷毀
3 )src/database/mongoose/mongoose.constants.ts
// 這個配置模擬調接口/讀數據庫獲取的
export const tenantMap = new Map([['1', 'T1'],['2', 'T2']
]);
export const defaultTenant = tenantMap.values().next().value; // 第一個// 新增如下
export const DEFAULT_DB_CONNECTION = 'DatabaseConnection';
export const MONGOOSE_MODULE_OPTIONS = 'MongooseModuleOptions';
export const MONGOOSE_CONNECTION_NAME = 'MongooseConnectionName';export const RAW_OBJECT_DEFINITION = 'RAW_OBJECT_DEFINITION';
3 )src/database/mongoose/mongoose.utils.ts
import { Logger } from '@nestjs/common';
import { Observable } from 'rxjs';
import { delay, retryWhen, scan } from 'rxjs/operators';export function handleRetry(retryAttempts = 9,retryDelay = 3000,verboseRetryLog = false,
): <T>(source: Observable<T>) => Observable<T> {const logger = new Logger('MongooseModule');return <T>(source: Observable<T>) =>source.pipe(retryWhen((e) =>e.pipe(scan((errorCount, error) => {const verboseMessage = verboseRetryLog? ` Message: ${error.message}.`: '';const retryMessage =retryAttempts > 0 ? ` Retrying (${errorCount + 1})...` : '';logger.error(['Unable to connect to the database.',verboseMessage,retryMessage,].join(''),error.stack,);if (errorCount + 1 >= retryAttempts) {throw error;}return errorCount + 1;}, 0),delay(retryDelay),),),);
}
- 工具包的部分函數
4 )測試
- 重啟項目,連入其中一個mongo 的 docker 容器,進入數據庫,執行
db.serverStatus().connections
- 調用對應數據庫的租戶標識的接口,再次執行
db.serverStatus().connections
- 可看到
current
和active
增加了,后面多次調用同一租戶接口則不會再增加 - 結束程序,則會銷毀客戶端,相應數字會同步減少
- 這樣就完成了相關功能
總結
- 問題本質:Mongoose 在 NestJS 中的連接未復用,導致連接數異常增長
- 核心影響:IO 資源浪費、性能下降、數據庫連接池耗盡
- 解決思路:
- 在服務層緩存已有連接
- 修改 Mongoose 模塊源碼邏輯
- 使用第三方連接池庫進行封裝
- 最佳實踐:
- 多租戶系統中應確保數據庫連接的唯一性與復用性
- 建議對 Mongoose 模塊進行輕量級封裝以適配業務需求