其实别人已经写得很好了,但是我将几个需要注意的点给标注一下。

需求:及时得到线上用户nginx访问日志分析统计结果!我需要知道IP访问排行、链接访问排行、静态资源、404错误排行等
直接上效果图:

具体安装步骤如下:
1、安装依赖

[root@iZbp1f0xuq9rc41s6gdvfyZ ~]# mkdir access 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# yum install glib2 glib2-devel GeoIP-devel  ncurses-devel zlib zlib-devel -y 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://github.com/maxmind/geoip-api-c/releases/download/v1.6.11/GeoIP-1.6.11.tar.gz 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -zvxf GeoIP-1.6.11.tar.gz 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd GeoIP-1.6.11 
[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# ./configure 
[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# make && make install 

2、安装goaccess

[root@iZbp1f0xuq9rc41s6gdvfyZ GeoIP-1.6.11]# cd .. 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# wget https://tar.goaccess.io/goaccess-1.3.tar.gz 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# tar -xzvf goaccess-1.3.tar.gz 
[root@iZbp1f0xuq9rc41s6gdvfyZ access]# cd goaccess-1.3 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./configure --enable-utf8 --enable-geoip=legacy 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# make && make install

3、校对nginx和goaccess日志格式
查看当前的nginx配置,日志格式

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd 
/usr/local/nginx/conf 
[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# cat /usr/local/nginx/conf/nginx.conf

记住这一段,后面有用。每个机器配置可能不一样哦,使用你自己的

log_format  main  '$remote_addr - $remote_user [$time_local] "$request" ' 
		'$status $body_bytes_sent "$http_referer" '  
		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"'; 

4、goaccess日志格式配置

一:使用nginx2goaccess.sh脚本将nginx日志格式格式化为goaccess能识别的日志格式,nginx2goaccess.sh脚本内容在 

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# vi nginx2goaccess.sh 
 
#!/bin/bash 
# 
# Convert from this: 
#   http://nginx.org/en/docs/http/ngx_http_log_module.html 
# To this: 
#   https://goaccess.io/man 
# 
# Conversion table: 
#   $time_local         %d:%t %^ 
#   $host               %v 
#   $http_host          %v 
#   $remote_addr        %h 
#   $request_time       %T 
#   $request_method     %m 
#   $request_uri        %U 
#   $server_protocol    %H 
#   $request            %r 
#   $status             %s 
#   $body_bytes_sent    %b 
#   $bytes_sent         %b 
#   $http_referer       %R 
#   $http_user_agent    %u 
# 
# Samples: 
# 
# log_format combined '$remote_addr - $remote_user [$time_local] ' 
# '"$request" $status $body_bytes_sent ' 
# '"$http_referer" "$http_user_agent"'; 
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent"' 
# 
# log_format compression '$remote_addr - $remote_user [$time_local] ' 
# '"$request" $status $bytes_sent ' 
# '"$http_referer" "$http_user_agent" "$gzip_ratio"'; 
#   ./nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $bytes_sent "$http_referer" "$http_user_agent" "$gzip_ratio"' 
# 
# log_format main 
# '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\t' 
# 'Local:\t$status\t$body_bytes_sent\t$request_time\t' 
# 'Proxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\t' 
# 'Agent:\t$http_user_agent\t' 
# 'Fwd:\t$http_x_forwarded_for'; 
#   ./nginx2goaccess.sh '$remote_addr\t$time_local\t$host\t$request\t$http_referer\t$http_x_mobile_group\tLocal:\t$status\t$body_bytes_sent\t$request_time\tProxy:\t$upstream_cache_status\t$upstream_status\t$upstream_response_length\t$upstream_response_time\tAgent:\t$http_user_agent\tFwd:\t$http_x_forwarded_for' 
# 
# log_format main 
# '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t' 
# '${http_referer}\t${http_x_mobile_group}\t' 
# 'Local:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\t' 
# 'Proxy:\t${upstream_status}\t${upstream_cache_status}\t' 
# '${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\t' 
# 'Agent:\t${http_user_agent}\t' 
# 'Fwd:\t${http_x_forwarded_for}'; 
#   ./nginx2goaccess.sh '${time_local}\t${remote_addr}\t${host}\t${request_method}\t${request_uri}\t${server_protocol}\t${http_referer}\t${http_x_mobile_group}\tLocal:\t${status}\t*${connection}\t${body_bytes_sent}\t${request_time}\tProxy:\t${upstream_status}\t${upstream_cache_status}\t${upstream_response_length}\t${upstream_response_time}\t${uri}${log_args}\tAgent:\t${http_user_agent}\tFwd:\t${http_x_forwarded_for}' 
# 
# Author: Rogério Carvalho Schneider <stockrt@gmail.com> 
 
# Params 
log_format="$1" 
 
# Usage 
if [[ -z "$log_format" ]]; then 
    echo "Usage: $0 '<log_format>'" 
    exit 1 
fi 
 
# Variables map 
conversion_table="time_local,%d:%t_%^ 
host,%v 
http_host,%v 
remote_addr,%h 
request_time,%T 
request_method,%m 
request_uri,%U 
server_protocol,%H 
request,%r 
status,%s 
body_bytes_sent,%b 
bytes_sent,%b 
http_referer,%R 
http_user_agent,%u" 
 
# Conversion 
for item in $conversion_table; do 
    nginx_var=${item%%,*} 
    goaccess_var=${item##*,} 
    goaccess_var=${goaccess_var//_/ } 
    log_format=${log_format//\$\{$nginx_var\}/$goaccess_var} 
    log_format=${log_format//\$$nginx_var/$goaccess_var} 
done 
log_format=$(echo "$log_format" | sed 's/${[a-z_]*}/%^/g') 
log_format=$(echo "$log_format" | sed 's/$[a-z_]*/%^/g') 
 
# Config output 
echo " 
- Generated goaccess config: 
time-format %T 
date-format %d/%b/%Y 
log_format $log_format 
" 
 
# EOF 

二:使用nginx2goaccess.sh方式获取日志格式

将第二步里面Nginx日志的单引号删除: 
 
删除前:'$remote_addr - $remote_user [$time_local] "$request" ' 
		'$status $body_bytes_sent "$http_referer" '  
		'"$http_user_agent" "$http_x_forwarded_for" "$request_body"' 
 
删除后:$remote_addr - $remote_user [$time_local] "$request"  
		$status $body_bytes_sent "$http_referer"  
		"$http_user_agent" "$http_x_forwarded_for" "$request_body" 
 
然后执行命令如: 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '删除后' 
执行真实命令: 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# sh nginx2goaccess.sh '$remote_addr - $remote_user [$time_local] "$request" $status $body_bytes_sent "$http_referer" "$http_user_agent" "$http_x_forwarded_for" "$request_body"' 

得到如下结果:

 
- Generated goaccess config: 
time-format %T 
date-format %d/%b/%Y 
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body" 

三:将上面生成的格式在goaccess-1.3/config下面创建一个nginxlog.conf

[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd config 
 
[root@iZbp1f0xuq9rc41s6gdvfyZ config]# vi nginxlog.conf 
 
# Generated goaccess config: 
time-format %T 
date-format %d/%b/%Y 
log_format %h - %^ [%d:%t %^] "%r" %s %b "%R" "%u" "%^" "%r_body"

5、配置完成,开始见证奇迹的时刻,生成统计页面。

第一步:生成结果文件夹的位置   /data/nginx 
[root@iZbp1f0xuq9rc41s6gdvfyZ /]# mkdir /data 
[root@iZbp1f0xuq9rc41s6gdvfyZ /]# cd data/ 
[root@iZbp1f0xuq9rc41s6gdvfyZ data]# mkdir nginx 
 
第二步:nginx日志位置、最好做了每日切割,否则太大了 
[root@iZbp1f0xuq9rc41s6gdvfyZ logs]# pwd 
/usr/local/nginx/logs 
 
第三步:开始日志分析,需要回到goaccess-1.3 文件夹 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# ./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html 
 
第四步:生成中文的方法 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# LANG="zh_CN.UTF-8" bash -c "./goaccess -f /usr/local/nginx/logs/access.log -p config/nginxlog.conf -o /data/nginx/report.html" 
 
说明:第三步、第四步分别生成的是当前时间的日志分析结果,五分钟之后想看新的得再执行一次 
下面我们写个定时器,让五分钟分析一次 
 
下面配置定时更新 
第五步:创建脚本 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# mkdir scripts 
[root@iZbp1f0xuq9rc41s6gdvfyZ goaccess-1.3]# cd scripts 
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# vi goaccess.sh 
 
#!/bin/bash 
LANG="zh_CN.UTF-8"           #页面转换为中文 
"/root/goaccess-1.3/goaccess -f /usr/local/nginx/logs/access.log -p /root/goaccess-1.3/config/nginxlog.conf -o /data/nginx/report.html" 
 
第六步:加入定时器 
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# chmod + ./goaccess.sh 
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# pwd 
/root/goaccess-1.3/scripts 
[root@iZbp1f0xuq9rc41s6gdvfyZ scripts]# crontab -e 
# 定时goaccess 每五分钟生成的信息保存为html 
5 * * * * /root/goaccess-1.3/scripts/goaccess.sh 

6、配置nginx的配置文件,可访问

配置nginx访问刚刚生成的report.html页面,阿里云的朋友自行配置即可,没多大的区别!

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# pwd 
/usr/local/nginx/conf 
 
[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# vi nginx.conf 
加上此段: 
 
server{ 
	listen 8080; 
	server_name localhost; 
 
	location /report.html { 
            alias /data/nginx/report.html; 
        }  
} 

重启nginx

[root@iZbp1f0xuq9rc41s6gdvfyZ conf]# /usr/local/nginx/sbin/nginx -c /usr/local/nginx/conf/nginx.conf 

在浏览器就 IT虾米网 可以浏览上上面的截图了。


评论关闭
IT源码网

微信公众号号:IT虾米 (左侧二维码扫一扫)欢迎添加!

Tesseract5.0 图像识别本地服务 三 【训练自定义字库,提高图片的识别效果】