1、Freeke 2008-12-10Nginx 反向代理+Varnish 技術實現目標：一、 Nginx 反向代理1、 安裝條件：Nginx: http:/sysoev.ru/nginx/nginx-0.6.32.tar.gzSSL: /source/openssl-0.9.8g.tar.gzPcre: ftp:/ftp.csx.cam.ac.uk/pub/software/programming/pcre/pcre-7.7.tar.gzZlib: 2、 安裝：l Ssl安裝：rootChinarenservice oracle# tar -zxvf o

2、penssl-0.9.8g.tar.gzrootChinarenservice oracle# cd openssl-0.9.8grootChinarenservice openssl-0.9.8g# ./config -prefix=/usr/local/openssl/rootChinarenservice openssl-0.9.8g# makerootChinarenservice openssl-0.9.8g# make installl Pcre 安裝：rootChinarenservice oracle# tar -zxvf pcre-7.7.tar.gzrootChinaren

3、service oracle# cd pcre-7.7rootChinarenservice pcre-7.7# ./configure -prefix=/usr/local/pcrerootChinarenservice pcre-7.7# makerootChinarenservice pcre-7.7# make installMake 時報錯：libtool: ignoring unknown tag CXXlibtool: unrecognized option -DHAVE_CONFIG_H'Try libtool -help' for more informati

4、on.make1: * pcrecpp.lo Error 1make1: Leaving directory /home/beijing/pcre-7.7'make: * all Error 2原因：pcre-7.7 configuration summary: Install prefix . : /usr/local/pcre C preprocessor . : gcc -E C compiler . : gcc C+ preprocessor . : C+ compiler . : Linker . : /usr/bin/ld C preprocessor flags . :

5、C compiler flags . : -O2 C+ compiler flags . : Linker flags . : Extra libraries . :沒有裝GCC C+包：gcc-c+-3.4.6-8.i386.rpm libstdc+-devel-3.4.6-8.i386.rpml Zlib 安裝：rootChinarenservice oracle# tar -zxvf zlib-1.2.3.tar.gzrootChinarenservice oracle# cd zlib-1.2.3rootChinarenservice zlib-1.2.3# ./configure -

6、prefix=/usr/local/zlibrootChinarenservice zlib-1.2.3# makerootChinarenservice zlib-1.2.3# make installl Tcmalloc 安裝：rootchinarenservice # wget /releases/libunwind/libunwind-0.99-alpha.tar.gzrootchinarenservice # tar zxvf libunwind-0.99-alpha.tar.gzrootchinarenservice #

7、 cd libunwind-0.99-alpha/rootchinarenservice # CFLAGS=-fPIC ./configurerootchinarenservice # make CFLAGS=-fPICrootchinarenservice # make CFLAGS=-fPIC installrootchinarenservice # wget http:/google-rootchinarenservice # tar zxvf google-perftools-0.98.tar.gzrootchinarenservice # cd google-perftools-0.

8、98/rootchinarenservice # ./configurerootchinarenservice # make && make installrootchinarenservice # echo "/usr/local/lib" > /etc/ld.so.conf.d/usr_local_lib.confrootchinarenservice # ldconfigroot chinarenservice local# lsof -n | grep tcmallocnginx 7323 root mem REG 8,2 1412859 44

9、0730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7324 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7325 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7326 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7327 www mem REG 8,2 14

10、12859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7328 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7329 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7330 www mem REG 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0nginx 7331 www mem RE

11、G 8,2 1412859 440730 /usr/local/lib/libtcmalloc.so.0.0.0l Nginx 安裝：rootChinarenservice oracle# tar -zxvf nginx-0.6.32.tar.gzrootChinarenservice oracle# cd nginx-0.6.32rootChinarenservice nginx-0.6.32# ./configure -prefix=/usr/local/nginx -with-http_ssl_module -with-pcre=/root/pcre-7.7 -with-zlib=/ro

12、ot/zlib-1.2.3 -with-http_stub_status_module -with-http_realip_module -with-http_addition_module -with-http_sub_module -with-http_dav_module -with-http_flv_module -with-openssl=/root/openssl-0.9.8g -with-google_perftools_modulerootChinarenservice nginx-0.6.32# makerootChinarenservice nginx-0.6.32# ma

13、ke install./configure -with-http_ssl_module -with-pcre=./pcre-7.8 -with-zlib=./zlib-1.2.3 -with-http_stub_status_module -with-openssl=./openssl-0.9.8g3、 配置：rootChinarenservice oracle# cat /usr/local/nginx/conf/nginx.confuser nobody nobody;worker_processes 30;error_log logs/error.log notice;pid logs/

14、nginx.pid;events use epoll; worker_connections 40960;http include mime.types;default_type application/octet-stream;log_format main '$remote_addr - $remote_user $time_local ' '"$request" $status $bytes_sent ' '"$http_referer" "$http_user_agent" '

15、'"$gzip_ratio"'keepalive_timeout 150;server_names_hash_bucket_size 64;upstream cache ip_hash;server 66:8080; /varnish server 1server ; /apache server server listen :80;server_name ;access_log logs/.access.log main;location / proxy_pass http:/cache;

16、proxy_redirect /; proxy_set_header Host $host; proxy_set_header X-Real-IP $remote_addr; proxy_set_header X-Forwarded-For $proxy_add_x_forwarded_for; proxy_set_header X-Is-EDU 0;client_max_body_size 50m;                &#

17、160;  client_body_buffer_size 256k;                   proxy_connect_timeout 10;                   proxy_send_timeou

18、t 15;                   proxy_read_timeout 15;                   proxy_buffer_size 4k;     

19、0;             proxy_buffers 4 32k;                   proxy_busy_buffers_size 64k;           &#

20、160;       proxy_temp_file_write_size 64k; server listen :81;server_name ;location /NginxStatus stub_status on;access_log off;allow 47;allow;deny all; 以上配置為nginx 做反向代理，監聽:80的IP，接收 的域名請求，轉發到后端varnish緩存服務器4、 優化：l 修改open

21、files數顯示open files數rootChinarenservice oracle# ulimit -acore file size (blocks, -c) 0data seg size (kbytes, -d) unlimitedfile size (blocks, -f) unlimitedpending signals (-i) 1024max locked memory (kbytes, -l) 32max memory size (kbytes, -m) unlimitedopen files (-n) 1024修改open files數rootChinarenservic

22、e oracle# ulimit -n 8192l 優化Linux內核參數rootChinarenservice oracle# vi /etc/sysctl.conf在末尾增加以下內容：net.ipv4.tcp_fin_timeout = 30net.ipv4.tcp_keepalive_time = 300net.ipv4.tcp_syncookies = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.ip_local_port_range = 5000 65000使配置立即生效：rootChinarenserv

23、ice oracle# /sbin/sysctl -pnet.ipv4.tcp_fin_timeout = 30#表示如果套接字由本端要求關閉，這個參數決定了它保持在FIN-WAIT-2狀態的時間。net.ipv4.tcp_keepalive_time = 300#表示當Keepalive起用的時候，TCP發送keepalive消息的頻繁度。默認值是2小時，這里我改為5分鐘。net.ipv4.tcp_syncookies = 1#表示開啟SYN Cookies。當出現SYN等待隊列溢出時，啟用cookies來處理，可防范少量SYN攻擊，預設為0，表示關閉，我將它開啟。net.ipv4.tcp_

24、tw_reuse = 1#表示開啟重用。允許將TIME-WAIT sockets重新用于新的TCP連接，預設為0，表示關閉，我將它開啟。net.ipv4.tcp_tw_recycle = 1#表示開啟TCP連接中TIME-WAIT sockets的快速回收，預設為0，表示關閉，我將它開啟。net.ipv4.ip_local_port_range = 5000 65000#表示用來向外連接的端口范圍。預設情況下很小：32768到61000，我將它改為5000到65000。net.ipv4.tcp_max_syn_backlog = 8192#表示SYN隊列的長度，預設為1024，加大隊列長度為8

25、192，可容納更多等待連接的網絡連接數。net.ipv4.tcp_max_tw_buckets = 5000#表示系統同時保持TIME_WAIT套接字的最大數量，如果超過這個數字，TIME_WAIT套接字將立刻被清除并印出警告訊息。預設是180000，改為5000。之后記得執行 /sbin/sysctl -p 使配置生效。l 不停止Nginx服務的情況下平滑變更Nginx配置rootChinarenservice oracle# kill -HUP cat /usr/local/nginx/logs/nginx.pidl5、 日志處理由于在Nginx上代理的網站訪問量非常大，每天的日志達到1G

26、以上，如果不對日志分天處理，那日志文件將非常大，并且不方便進行日志分析。l Logrotate/home/log/*access.log daily missingok rotate 31 nocompress prerotate #/usr/local/awstats/wwwroot/cgi-bin/awstats.pl -update -config=shutter endscript postrotate if -f /usr/local/nginx/logs/nginx.pid ; then kill -USR1 cat /usr/local/nginx/logs/nginx.pid

27、fi endscript l Shell寫一個shell，每天在晚上11點59分執行將日志文件分天處理：rootChinarenservice log# crontab l59 23 * * * /var/InfiNET/usemon/logcronnginx.shrootChinarenservice usemon# less logcronnginx.sh#!/bin/sh/bin/mv /home/log/bbs.access.log /home/log/bbs.access.date +%Y%m%d.log/bin/mv /home/log/register.access.log /h

28、ome/log/register.access.date +%Y%m%d.log/bin/mv /home/log/shutter.access.log /home/log/shutter.access.date +%Y%m%d.log#/bin/mv /home/log/bbs.access.log /home/log/bbs.access.%YYYY-24-%MM-24-%DD-24.log#/bin/mv /home/log/register.access.log /home/log/register.access.%YYYY-24-%MM-24-%DD-24.log#/bin/mv /

29、home/log/shutter.access.log /home/log/shutter.access.%YYYY-24-%MM-24-%DD-24.log/usr/bin/killall -s USR1 nginxl6、 awstats日志分析請參見Freeke Nginx Awstats Install7、二、 Varnish 緩存Varnish優點：1、Varnish采用了“Visual Page Cache”技術，在內存的利用上，Varnish比Squid具有優勢，它避免了Squid頻繁在內存、磁盤中交換文件，性能要比Squid高。2、Varnish的穩定性非常好3、通過Varnis

30、h管理端口，可以使用正則表達式快速、批量地清除部分緩存，這一點是Squid不能具備的。Varnish網站緩存加速器安裝：1、創建www用戶和組，以及Varnish緩存文件存放目錄（/var/InfiNET/cache）：rootChinarenservice oracle# /usr/sbin/groupadd www -g 48rootChinarenservice oracle# /usr/sbin/useradd -u 48 -g www wwwrootChinarenservice oracle# mkdir -p /var/InfiNET/cacherootChinarenservi

31、ce oracle# chmod +w /var/InfiNET/cacherootChinarenservice oracle# chown -R www:www /var/InfiNET/cache2、創建Varnish日志目錄（/var/logs/）：rootChinarenservice oracle# mkdir -p /usr/local/varnish/logsrootChinarenservice oracle# chmod +w /usr/local/varnish/logsrootChinarenservice oracle# chown -R www:www /usr/l

32、ocal/varnish/logs3、編譯安裝varnish：下載：rootChinarenservice oracle# wget rootChinarenservice oracle# tar zxvf varnish-1.1.2.tar.gzrootChinarenservice oracle# cd varnish-1.1.2rootChinarenservice oracle# ./configure -prefix=/usr/local/varnishrootChinarenservice oracle# make && make install./configur

33、e -enable-debugging-symbols -enable-developer-warnings -enable-dependency-tracking -prefix=/usr/local/varnish注意,我在進行make步驟時,出現如下錯誤: "varnishhist.c:35:20: error: curses.h: No such file or directory" 造成該問題的原因是因為系統中少了ncurses-devel包4、創建Varnish配置文件：rootChinarenservice oracle# vi /usr/local/varn

34、ish/vcl.confbackend mblog set backend.host = "68" set backend.port = "80" acl purge "localhost" "" ""/24; ""/24;sub vcl_recv if (req.http.User-Agent "(Baiduspider|google)")   

35、0;                     error 405 "Not allowed."                 if (req.http.Referer "(|2

36、|7878758.com|||||7878758.com||||||||72|4)")                      &

37、#160;  error 405 "Not allowed."        if (req.request = "PURGE") if (!client.ip purge) error 405 "Not allowed." lookup; if (req.http.host "") set req.backend = mblog; if (req.request != "GET" && req.re

38、quest != "HEAD") pipe; else lookup; else error 404 "Koncept Cache Server" lookup; sub vcl_hit if (req.request = "PURGE") set obj.ttl = 0s; error 200 "Purged." sub vcl_miss if (req.request = "PURGE") error 404 "Not in cache." sub vcl_fetch i

39、f (req.request = "GET" && req.url ".(txt|js|gif|jpg|jpeg|tom|swf|css)$") set obj.ttl = 3600s; else set obj.ttl = 30d; 對以上配置文件解釋一下：(1)、Varnish通過反向代理請求后端IP為，端口為80的apache服務器；(2)、Varnish允許localhost、、10.167.0.*源IP通過PURGE方法清除緩存；(3)、Varnish對域名為的請求進行處理，非域名的請求則

40、返回“freeke Cache Server”；(4)、Varnish對HTTP協議中的GET、HEAD請求進行緩存，對POST請求透過，讓其直接訪問后端Web服務器。之所以這樣配置，是因為POST請求一般是發送數據給服務器的，需要服務器接收、處理，所以不緩存；(5)、Varnish對以.txt和.js等結尾的URL緩存時間設置1小時，對其它的URL緩存時間設置為30天。5、啟動VarnishrootChinarenservice oracle# limit -SHn 51200rootChinarenservice oracle# /usr/local/varnish/sbin/varnis

41、hd -n /var/InfiNET/cache -f /usr/local/varnish/vcl.conf -a :8080 -s file,/var/InfiNET/cache/varnish_cache.data,2G -g www -u www -w 30000,51200,10 -T :10030 -p client_http11=on -P /usr/local/varnish/var/varnish.pid6、啟動varnishncsa用來將Varnish訪問日志寫入日志文件：rootChinarenservice oracle# /usr/lo

42、cal/varnish/bin/varnishncsa -n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log &7、配置開機自動啟動VarnishrootChinarenservice oracle# vi /etc/rc.localulimit -SHn 51200/usr/local/varnish/sbin/varnishd -n /var/InfiNET/cache -f /usr/local/varnish/vcl.conf -a :8080 -s file,/var/InfiNET/cache

43、/varnish_cache.data,2G -g www -u www -w 30000,51200,10 -T :10030 -p client_http11=on -P /usr/local/varnish/var/varnish.pid/usr/local/varnish/bin/varnishncsa -n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log&8、優化Linux內核參數rootChinarenservice oracle# vi /etc/sysctl.confnet.ipv4.

44、tcp_fin_timeout = 30net.ipv4.tcp_keepalive_time = 300net.ipv4.tcp_syncookies = 1net.ipv4.tcp_tw_reuse = 1net.ipv4.tcp_tw_recycle = 1net.ipv4.ip_local_port_range = 5000    65000再看看如何管理Varnish：1、查看Varnish服務器連接數與命中率：rootChinarenservice oracle# /usr/local/varnish/bin/varnishstat n /var/InfiNET

45、/cache/2、通過Varnish管理端口進行管理：用help看看可以使用哪些Varnish命令：rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 helpAvailable commands:ping timestampstatusstartstopstatsvcl.load vcl.inline vcl.use vcl.discard vcl.listvcl.show param.show -l param.set help commandurl.purge dump.pool3

46、、通過Varnish管理端口，使用正則表達式批量清除緩存：(1)、例：清除類似rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 url.purge /a/(2)、例：清除類似rootChinarenservice oracle# /usr/local/varnish/bin/varnishadm -T :3500 url.purge w*$(3)、例：清除所有緩存：rootChinarenservice oracle# /usr/local/varnish/bin/v

47、arnishadm -T :3500 url.purge *$4、一個清除Squid緩存的PHP函數（清除Varnish緩存同樣可以使用該函數，無需作任何修改，十分方便）：view plaincopy to clipboardprint?1. <?php   2. function purge($ip, $url)   3.   4.     $errstr = ''   5.  

48、60;  $errno = ''   6.     $fp = fsockopen ($ip, 80, $errno, $errstr, 2);   7.     if (!$fp)   8.        9.     &

49、#160;    return false;   10.        11.     else  12.        13.         $out = "PURGE $url HTTP/1.1rn" &

50、#160; 14.         $out .= "Host:rn"   15.         $out .= "Connection: closernrn"   16.         fputs ($f

51、p, $out);   17.         $out = fgets($fp , 4096);   18.         fclose ($fp);   19.         return true;  

52、0;20.        21.   22.   23. purge("", "/index.php");   24. ?>  附1：Varnish官方網站/附2：日志處理2007年12月10日，我寫了一個每天0點運行，按天切割Varnish日志，生成一個壓縮文件，同時刪除上個月舊日志的腳本（/usr/local/varnish

53、/logs/cutlog.sh）：rootChinarenservice oracle# /usr/local/varnish/logs/cutlog.sh #!/bin/sh# This file run at 00:00date=$(date -d "yesterday" +"%Y-%m-%d")pkill -9 varnishncsamv /usr/local/varnish/logs/youvideo.log /usr/local/varnish/logs/$date.log/usr/local/varnish/bin/varnishncsa -

54、n /var/InfiNET/cache -w /usr/local/varnish/logs/varnish.log &mkdir -p /usr/local/varnish/logs/youvideo/gzip -c /usr/local/varnish/logs/$date.log > /usr/local/varnish/logs/youvideo/$date.log.gzrm -f /usr/local/varnish/logs/$date.logrm -f /usr/local/varnish/logs/youvideo/$(date -d "-1 mont

55、h" +"%Y-%m*").log.gz設置在每天00:00定時執行： rootChinarenservice oracle# /usr/bin/crontab -e或者 rootChinarenservice oracle# vi /var/spool/cron/root0 0 * * * /bin/sh /usr/local/varnish/logs/cutlog.sh附3: varnish日志的rotaterootChinarenservice oracle# touch /etc/logrotate.d/varnishrootChinarenservice oracle# les /etc/logrotate.d/varnish/usr/local/varnish/logs/varnish.log     daily     rotate 60    &