go - aws-lambda 上的 sql.DB 连接太多

Question

正如我在 Golang 中的理解:DB 句柄意味着长期存在并在许多 goroutine 之间共享。

但是当我将 Golang 与 AWS lambda 结合使用时，情况就完全不同了，因为 lambda 会在函数完成时停止该函数。

我在 Lambda Invoke 函数中使用:defer db.Close() 但它不受影响。在 MySQL 上，它仍将连接保持为 Sleep query。结果，它导致 MySQL 上的太多连接。

目前，我必须将 MySQL 中的 wait_timeout 设置为较小的数字。但在我看来，这不是最好的解决方案。

在使用带有 Lambda 的 Go SQL 驱动程序时，有什么方法可以关闭连接吗？

谢谢，

Answer 1

有两个问题需要解决

• 正确管理 lambda 调用之间的状态
• 配置连接池

正确管理状态

让我们稍微了解一下 AWS 是如何管理容器的。来自AWS docs :

After a Lambda function is executed, AWS Lambda maintains the execution context for some time in anticipation of another Lambda function invocation. In effect, the service freezes the execution context after a Lambda function completes, and thaws the context for reuse, if AWS Lambda chooses to reuse the context when the Lambda function is invoked again. This execution context reuse approach has the following implications:

• Any declarations in your Lambda function code (outside the handler code, see Programming Model) remains initialized, providing additional optimization when the function is invoked again. For example, if your Lambda function establishes a database connection, instead of reestablishing the connection, the original connection is used in subsequent invocations. We suggest adding logic in your code to check if a connection exists before creating one.

• Each execution context provides 500MB of additional disk space in the /tmp directory. The directory content remains when the execution context is frozen, providing transient cache that can be used for multiple invocations. You can add extra code to check if the cache has the data that you stored. For information on deployment limits, see AWS Lambda Limits.

• Background processes or callbacks initiated by your Lambda function that did not complete when the function ended resume if AWS Lambda chooses to reuse the execution context. You should make sure any background processes or callbacks (in case of Node.js) in your code are complete before the code exits.

第一个要点表示状态在两次执行之间保持不变。让我们看看实际效果:

let counter = 0

module.exports.handler = (event, context, callback) => {
  counter++
  callback(null, { count: counter })
}

如果您部署它并连续调用多次，您会看到计数器会在两次调用之间递增。

现在您知道了 - 您不应该调用 defer db.Close() ，相反，您应该重用数据库实例。您可以通过简单地制作 db 来做到这一点包级变量。

首先，创建一个将导出 Open 的数据库包功能:

package database

import (
    "fmt"
    "os"

    _ "github.com/go-sql-driver/mysql"
    "github.com/jinzhu/gorm"
)

var (
    host = os.Getenv("DB_HOST")
    port = os.Getenv("DB_PORT")
    user = os.Getenv("DB_USER")
    name = os.Getenv("DB_NAME")
    pass = os.Getenv("DB_PASS")
)

func Open() (db *gorm.DB) {
    args := fmt.Sprintf("%s:%s@tcp(%s:%s)/%s?parseTime=true", user, pass, host, port, name)
    // Initialize a new db connection.
    db, err := gorm.Open("mysql", args)
    if err != nil {
        panic(err)
    }
    return
}

然后在你的 handler.go 文件中使用它:

package main

import (
    "context"

    "github.com/aws/aws-lambda-go/events"
    "github.com/aws/aws-lambda-go/lambda"
    "github.com/jinzhu/gorm"
    "github.com/<username>/<name-of-lib>/database"
)

var db *gorm.DB

func init() {
    db = database.Open()
}

func Handler() (events.APIGatewayProxyResponse, error) {
    // You can use db here.
    return events.APIGatewayProxyResponse{
        StatusCode: 201,
    }, nil
}

func main() {
    lambda.Start(Handler)
}

OBS:不要忘记替换 github.com/<username>/<name-of-lib>/database走正确的道路。

现在，您可能仍会看到 too many connections错误。如果发生这种情况，您将需要一个连接池。

配置连接池

来自 Wikipedia :

In software engineering, a connection pool is a cache of database connections maintained so that the connections can be reused when future requests to the database are required. Connection pools are used to enhance the performance of executing commands on a database.

您将需要一个连接池，允许的连接数必须等于并行运行的 lambda 数，您有两个选择:

• MySQL 代理

MySQL Proxy is a simple program that sits between your client and MySQL server(s) and that can monitor, analyze or transform their communication. Its flexibility allows for a wide variety of uses, including load balancing, failover, query analysis, query filtering and modification, and many more.

• AWS 极光:

Amazon Aurora Serverless is an on-demand, auto-scaling configuration for Amazon Aurora (MySQL-compatible edition), where the database will automatically start up, shut down, and scale capacity up or down based on your application's needs. It enables you to run your database in the cloud without managing any database instances. It's a simple, cost-effective option for infrequent, intermittent, or unpredictable workloads.

无论您如何选择，互联网上都有大量关于如何配置两者的教程。