我想将 Varnishlog 与 Logstash、Elasticsearch 和 Kibana 结合使用，以大致了解“我的服务器响应请求需要多长时间”等重要事项。 我的问题是，Varnish 将每个请求记录在多行中。
例如:
14 BackendOpen b default 127.0.0.1 42697 127.0.0.1 2727
14 BackendXID b 722117072
14 TxRequest b GET
14 TxURL b /images/maske.png
14 TxProtocol b HTTP/1.1
14 TxHeader b Host: ecomxenvm01:7272
14 TxHeader b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
14 TxHeader b Accept: image/png,image/*;q=0.8,*/*;q=0.5
14 TxHeader b Accept-Language: de,en-US;q=0.7,en;q=0.3
14 TxHeader b Referer: http://ecomxenvm01:7272/format.css
14 TxHeader b X-Varnish: 722117072
14 TxHeader b Accept-Encoding: gzip
14 RxProtocol b HTTP/1.1
14 RxStatus b 200
14 RxResponse b OK
14 RxHeader b Date: Thu, 02 Jul 2015 08:27:38 GMT
14 RxHeader b Server: Apache/2.2.15 (CentOS)
14 RxHeader b Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
14 RxHeader b ETag: "60010-111f8-5193f039fed80"
14 RxHeader b Accept-Ranges: bytes
14 RxHeader b Content-Length: 70136
14 RxHeader b Connection: close
14 RxHeader b Content-Type: image/png
14 Fetch_Body b 4(length) cls 0 mklen 1
14 Length b 70136
14 BackendClose b default
12 ReqStart c 172.16.145.134 57776 722117072
12 RxRequest c GET
12 RxURL c /images/maske.png
12 RxProtocol c HTTP/1.1
12 RxHeader c Host: ecomxenvm01:7272
12 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
12 RxHeader c Accept: image/png,image/*;q=0.8,*/*;q=0.5
12 RxHeader c Accept-Language: de,en-US;q=0.7,en;q=0.3
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c Referer: http://ecomxenvm01:7272/format.css
12 RxHeader c Connection: keep-alive
12 VCL_call c recv lookup
12 VCL_call c hash
12 Hash c /images/maske.png
12 Hash c ecomxenvm01:7272
12 VCL_return c hash
12 VCL_call c miss fetch
12 Backend c 14 default default
12 TTL c 722117072 RFC 120 -1 -1 1435825659 0 1435825658 0 0
12 VCL_call c fetch deliver
12 ObjProtocol c HTTP/1.1
12 ObjResponse c OK
12 ObjHeader c Date: Thu, 02 Jul 2015 08:27:38 GMT
12 ObjHeader c Server: Apache/2.2.15 (CentOS)
12 ObjHeader c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
12 ObjHeader c ETag: "60010-111f8-5193f039fed80"
12 ObjHeader c Content-Type: image/png
12 VCL_call c deliver deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 200
12 TxResponse c OK
12 TxHeader c Server: Apache/2.2.15 (CentOS)
12 TxHeader c Last-Modified: Wed, 24 Jun 2015 08:08:38 GMT
12 TxHeader c ETag: "60010-111f8-5193f039fed80"
12 TxHeader c Content-Type: image/png
12 TxHeader c Content-Length: 70136
12 TxHeader c Accept-Ranges: bytes
12 TxHeader c Date: Thu, 02 Jul 2015 08:27:38 GMT
12 TxHeader c X-Varnish: 722117072
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 Length c 70136
12 ReqEnd c 722117072 1435825658.602955103 1435825658.603794336 0.012183905 0.000789165 0.000050068
14 BackendOpen b default 127.0.0.1 42698 127.0.0.1 2727
14 BackendXID b 722117073
14 TxRequest b GET
14 TxURL b /images/arrow.png
14 TxProtocol b HTTP/1.1
14 TxHeader b Host: ecomxenvm01:7272
14 TxHeader b User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
14 TxHeader b Accept: image/png,image/*;q=0.8,*/*;q=0.5
14 TxHeader b Accept-Language: de,en-US;q=0.7,en;q=0.3
14 TxHeader b Referer: http://ecomxenvm01:7272/format.css
14 TxHeader b X-Varnish: 722117073
14 TxHeader b Accept-Encoding: gzip
14 RxProtocol b HTTP/1.1
14 RxStatus b 200
14 RxResponse b OK
14 RxHeader b Date: Thu, 02 Jul 2015 08:27:38 GMT
14 RxHeader b Server: Apache/2.2.15 (CentOS)
14 RxHeader b Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
14 RxHeader b ETag: "60009-af-5193f0390ab40"
14 RxHeader b Accept-Ranges: bytes
14 RxHeader b Content-Length: 175
14 RxHeader b Connection: close
14 RxHeader b Content-Type: image/png
14 Fetch_Body b 4(length) cls 0 mklen 1
14 Length b 175
14 BackendClose b default
12 ReqStart c 172.16.145.134 57776 722117073
12 RxRequest c GET
12 RxURL c /images/arrow.png
12 RxProtocol c HTTP/1.1
12 RxHeader c Host: ecomxenvm01:7272
12 RxHeader c User-Agent: Mozilla/5.0 (Windows NT 6.1; rv:38.0) Gecko/20100101 Firefox/38.0
12 RxHeader c Accept: image/png,image/*;q=0.8,*/*;q=0.5
12 RxHeader c Accept-Language: de,en-US;q=0.7,en;q=0.3
12 RxHeader c Accept-Encoding: gzip, deflate
12 RxHeader c Referer: http://ecomxenvm01:7272/format.css
12 RxHeader c Connection: keep-alive
12 VCL_call c recv lookup
12 VCL_call c hash
12 Hash c /images/arrow.png
12 Hash c ecomxenvm01:7272
12 VCL_return c hash
12 VCL_call c miss fetch
12 Backend c 14 default default
12 TTL c 722117073 RFC 120 -1 -1 1435825659 0 1435825658 0 0
12 VCL_call c fetch deliver
12 ObjProtocol c HTTP/1.1
12 ObjResponse c OK
12 ObjHeader c Date: Thu, 02 Jul 2015 08:27:38 GMT
12 ObjHeader c Server: Apache/2.2.15 (CentOS)
12 ObjHeader c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
12 ObjHeader c ETag: "60009-af-5193f0390ab40"
12 ObjHeader c Content-Type: image/png
12 VCL_call c deliver deliver
12 TxProtocol c HTTP/1.1
12 TxStatus c 200
12 TxResponse c OK
12 TxHeader c Server: Apache/2.2.15 (CentOS)
12 TxHeader c Last-Modified: Wed, 24 Jun 2015 08:08:37 GMT
12 TxHeader c ETag: "60009-af-5193f0390ab40"
12 TxHeader c Content-Type: image/png
12 TxHeader c Content-Length: 175
12 TxHeader c Accept-Ranges: bytes
12 TxHeader c Date: Thu, 02 Jul 2015 08:27:38 GMT
12 TxHeader c X-Varnish: 722117073
12 TxHeader c Age: 0
12 TxHeader c Via: 1.1 varnish
12 TxHeader c Connection: keep-alive
12 Length c 175
12 ReqEnd c 722117073 1435825658.607635498 1435825658.608191013 0.003841162 0.000532389 0.000023127`
现在我的问题是:如何将这些行合并为一行？或者我如何总结这个请求？
我是否必须更改配置文件中的某些内容？或者有什么工具可以帮助我吗？
谢谢 厘米
配置文件:
DAEMON_OPTS="$DAEMON_OPTS -i RxStatus"
# Configuration file for varnish
#
# /etc/init.d/varnish expects the variable $DAEMON_OPTS to be set from this
# shell script fragment.
#
# Maximum number of open files (for ulimit -n)
NFILES=131072
# Locked shared memory (for ulimit -l)
# Default log size is 82MB + header
MEMLOCK=82000
# Maximum number of threads (for ulimit -u)
NPROCS="unlimited"
# Maximum size of corefile (for ulimit -c). Default in Fedora is 0
# DAEMON_COREFILE_LIMIT="unlimited"
# Set this to 1 to make init script reload try to switch vcl without restart.
# To make this work, you need to set the following variables
# explicit: VARNISH_VCL_CONF, VARNISH_ADMIN_LISTEN_ADDRESS,
# VARNISH_ADMIN_LISTEN_PORT, VARNISH_SECRET_FILE, or in short,
# use Alternative 3, Advanced configuration, below
RELOAD_VCL=1
# This file contains 4 alternatives, please use only one.
## Alternative 1, Minimal configuration, no VCL
#
# Listen on port 7272, administration on localhost:6082, and forward to
# content server on localhost:2727. Use a fixed-size cache file.
#
#DAEMON_OPTS="-a :6081 \
# -T localhost:6082 \
# -b localhost:8080 \
# -u varnish -g varnish \
# -s file,/var/lib/varnish/varnish_storage.bin,1G"
## Alternative 2, Configuration with VCL
#
# Listen on port 6081, administration on localhost:6082, and forward to
# one content server selected by the vcl file, based on the request. Use a
# fixed-size cache file.
#
#DAEMON_OPTS="-a :6081 \
# -T localhost:6082 \
# -f /etc/varnish/default.vcl \
# -u varnish -g varnish \
# -S /etc/varnish/secret \
# -s file,/var/lib/varnish/varnish_storage.bin,1G"
## Alternative 3, Advanced configuration
#
# See varnishd(1) for more information.
#
# # Main configuration file. You probably want to change it :)
VARNISH_VCL_CONF=/etc/varnish/default.vcl
#
# # Default address and port to bind to
# # Blank address means all IPv4 and IPv6 interfaces, otherwise specify
# # a host name, an IPv4 dotted quad, or an IPv6 address in brackets.
# VARNISH_LISTEN_ADDRESS=
VARNISH_LISTEN_PORT=7272
#
# # Telnet admin interface listen address and port
VARNISH_ADMIN_LISTEN_ADDRESS=127.0.0.1
VARNISH_ADMIN_LISTEN_PORT=6082
#
# # Shared secret file for admin interface
VARNISH_SECRET_FILE=/etc/varnish/secret
#
# # The minimum number of worker threads to start
VARNISH_MIN_THREADS=50
#
# # The Maximum number of worker threads to start
VARNISH_MAX_THREADS=1000
#
# # Idle timeout for worker threads
VARNISH_THREAD_TIMEOUT=120
#
# # Cache file location
VARNISH_STORAGE_FILE=/var/lib/varnish/varnish_storage.bin
#
# # Cache file size: in bytes, optionally using k / M / G / T suffix,
# # or in percentage of available disk space using the % suffix.
VARNISH_STORAGE_SIZE=1G
#
# # Backend storage specification
VARNISH_STORAGE="file,${VARNISH_STORAGE_FILE},${VARNISH_STORAGE_SIZE}"
#
# # Default TTL used when the backend does not specify one
VARNISH_TTL=120
#
# # DAEMON_OPTS is used by the init script. If you add or remove options, make
# # sure you update this section, too.
DAEMON_OPTS="-a ${VARNISH_LISTEN_ADDRESS}:${VARNISH_LISTEN_PORT} \
-f ${VARNISH_VCL_CONF} \
-T ${VARNISH_ADMIN_LISTEN_ADDRESS}:${VARNISH_ADMIN_LISTEN_PORT} \
-t ${VARNISH_TTL} \
-w ${VARNISH_MIN_THREADS},${VARNISH_MAX_THREADS},${VARNISH_THREAD_TIMEOUT} \
-u varnish -g varnish \
-S ${VARNISH_SECRET_FILE} \
-s ${VARNISH_STORAGE}"
#
## Alternative 4, Do It Yourself. See varnishd(1) for more information.
#
# DAEMON_OPTS=""
当我找到 Varnishlog 时:
[elk@ecomxenvm01 ~]$ sudo locate varnishlog
/etc/rc.d/init.d/varnishlog
/etc/rc.d/rc0.d/K10varnishlog
/etc/rc.d/rc1.d/K10varnishlog
/etc/rc.d/rc2.d/K10varnishlog
/etc/rc.d/rc3.d/K10varnishlog
/etc/rc.d/rc4.d/K10varnishlog
/etc/rc.d/rc5.d/K10varnishlog
/etc/rc.d/rc6.d/K10varnishlog
/usr/bin/varnishlog
/usr/share/man/man1/varnishlog.1.gz
/usr/share/man/man8/varnishlog_selinux.8.gz
最佳答案
你应该看看this .特别是选项 -I 和 -i。
带有标签的示例，仅显示 RxStatus 日志条目:
varnishlog -i RxStatus
使用正则表达式的示例，仅显示 ReqStart 和 ReqEnd 条目:
varnishlog -I "Req[Start|End]"
关于logstash - 如何获得总结的 Varnishlog，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/31179506/