我有多个客户端到单个服务器的双向 iperf 设置,用于网络监控。 iperf 服务器运行良好,并根据客户端编写的 cron 作业以 CSV 格式显示输出。
我希望编写一个 python 脚本来自动将这些 CSV 输出映射到 MySQL 数据库的过程;反过来,这些信息会定期更新和保存,而无需人工干预。
我正在使用 Ubuntu 13.10 机器作为 iperf 服务器。以下是我得到的示例 CSV 输出。这没有存储到文件中,只是显示在屏幕上。
s1:~$ iperf -s -y C
20140422105054,172.16.10.76,41065,172.16.10.65,5001,6,0.0-20.0,73138176,29215083
20140422105054,172.16.10.76,5001,172.16.10.65,56254,4,0.0-20.0,46350336,18502933
20140422105100,172.16.10.76,54550,172.16.10.50,5001,8,0.0-20.0,67895296,27129408
20140422105100,172.16.10.76,5001,172.16.10.50,58447,5,0.0-20.1,50937856,20292796
20140422105553,172.16.10.76,5001,172.16.10.65,47382,7,0.0-20.1,51118080,20358083
20140422105553,172.16.10.76,41067,172.16.10.65,5001,5,0.0-20.1,76677120,30524007
20140422105600,172.16.10.76,5001,172.16.10.50,40734,4,0.0-20.0,57606144,23001066
20140422105600,172.16.10.76,54552,172.16.10.50,5001,8,0.0-20.0,70123520,28019115
20140422110053,172.16.10.76,41070,172.16.10.65,5001,5,0.0-20.1,63438848,25284066
20140422110053,172.16.10.76,5001,172.16.10.65,46462,6,0.0-20.1,11321344,4497094
我要映射到的字段是:timestamp, server_ip, server_port, client_ip, client_port, tag_id, interval, transferred, bandwidth
我想定期将此 CSV 输出映射到 MySQL 数据库,为此我知道我必须编写一个 Python 脚本(在 cron 作业中)查询和存储在 MySQL 数据库中。我是 Python 脚本和数据库查询的初学者。
我在 [ https://serverfault.com/questions/566737/iperf-csv-output-format] 上进行了关于服务器故障的另一次讨论。 ;并希望基于此构建我的查询。
最佳答案
生成SQL脚本,然后运行
如果您不想使用像 sqlalchemy
这样的复杂解决方案,可以采用以下方法。
- 拥有您的 csv 数据,将它们转换为 SQL 脚本
- 使用mysql命令行工具运行这个脚本
在你第一次这样做之前,确保你在数据库中创建了所需的数据库结构(我留给你)。
我的以下示例使用(只是为了方便起见)包 docopt
,因此您需要安装它:
$ pip install docopt
CSV 到 SQL 脚本的转换实用程序
csv2sql.py
:
"""
Usage:
csv2sql.py [--table <tablename>] <csvfile>
Options:
--table <tablename> Name of table in database to import into [default: mytable]
Convert csv file with iperf data into sql script for importing
those data into MySQL database.
"""
from csv import DictReader
from docopt import docopt
if __name__ == "__main__":
args = docopt(__doc__)
fname = args["<csvfile>"]
tablename = args["--table"]
headers = ["timestamp",
"server_ip",
"server_port",
"client_ip",
"client_port",
"tag_id",
"interval",
"transferred",
"bandwidth"
]
sql = """insert into {tablename}
values ({timestamp},"{server_ip}",{server_port},"{client_ip}",{client_port},{tag_id},"{interval}",{transferred},{bandwidth});"""
with open(fname) as f:
reader = DictReader(f, headers, delimiter=",")
for rec in reader:
print(sql.format(tablename=tablename, **rec)) # python <= 2.6 will fail here
将 CSV 转换为 SQL 脚本
先介绍一下转换工具:
$ python csv2sql.py -h
Usage:
csv2sql.py [--table <tablename>] <csvfile>
Options:
--table <tablename> Name of table in database to import into [default: mytable]
Convert csv file with iperf data into sql script for importing
those data into MySQL database.
将数据保存在文件 data.csv
中:
$ python csv2sql.py data.csv
insert into mytable
values (20140422105054,"172.16.10.76",41065,"172.16.10.65",5001,6,"0.0-20.0",73138176,29215083);
insert into mytable
values (20140422105054,"172.16.10.76",5001,"172.16.10.65",56254,4,"0.0-20.0",46350336,18502933);
insert into mytable
values (20140422105100,"172.16.10.76",54550,"172.16.10.50",5001,8,"0.0-20.0",67895296,27129408);
insert into mytable
values (20140422105100,"172.16.10.76",5001,"172.16.10.50",58447,5,"0.0-20.1",50937856,20292796);
insert into mytable
values (20140422105553,"172.16.10.76",5001,"172.16.10.65",47382,7,"0.0-20.1",51118080,20358083);
insert into mytable
values (20140422105553,"172.16.10.76",41067,"172.16.10.65",5001,5,"0.0-20.1",76677120,30524007);
insert into mytable
values (20140422105600,"172.16.10.76",5001,"172.16.10.50",40734,4,"0.0-20.0",57606144,23001066);
insert into mytable
values (20140422105600,"172.16.10.76",54552,"172.16.10.50",5001,8,"0.0-20.0",70123520,28019115);
insert into mytable
values (20140422110053,"172.16.10.76",41070,"172.16.10.65",5001,5,"0.0-20.1",63438848,25284066);
insert into mytable
values (20140422110053,"172.16.10.76",5001,"172.16.10.65",46462,6,"0.0-20.1",11321344,4497094);
将其全部放入文件data.sql
:
$ python csv2sql.py data.csv > data.sql
将 data.sql
应用到您的 MySQL 数据库
最后使用 mysql
命令(由 MySQL 提供)导入数据库:
$ myslq --user username --password password db_name < data.sql
关于python - 使用 Python 将 CSV 输出导出并映射到 MySQL 表,我们在Stack Overflow上找到一个类似的问题: https://stackoverflow.com/questions/23300645/