1. -QPSRTCPU
2. 12QPS/RT---(CPU/IO)34567FULL GC89 3. QPSQuery-per-second1RTResponse-time1 4. QPS1.2.QPS11 Detail 236 ShopSystem 115QPSDetail118ShopSystem58detailshopsystemRT1.2.QPS 5. hesperCPUIO200100ms1IO 11 12 32 22QP 33QPS45491searchAuction.vmQPS2QPS49190100ms 6. QPS1QPSQPSQPS1hesper2QPSQPS=1000/RTQPSCPUQPS=1000/RTQPS 7. 1=((+cpu)/cpu) * cpu1QPSQPS23CPUIO 8. 1 2:=((+cpu)/cpu) *cpu3CPU 9. -detailapache3000tomcat200CPU10CPU 85% 60(Full GC)CPU10200 10. 2 = / * QPSQPS11001100 21Mqps1100rt1.911ms ab -n10000 -c2 http://192.168.211.1:8080/test.jsp?thm=1024000 1001Mqps1100rt89ms ab -n10000 -c100 http://192.168.211.1:8080/test.jsp?thm=1024000 2001Mqps1100rt170ms ab -n100000 -c200 http://192.168.211.1:8080/test.jsp?thm=1024000 11. IOCPU10ms + IO 40ms = 50msCPU5ms45ms50ms45msqps2022CPUQPS1IO40ms80ms90ms50ms90msQPS 12. QPSRT1webRTCPUIOsleepwaithesper io-cpu-io-wait-cpu-ioQPSRT3QPS=1000ms/RT1QPS QPSRT RT1 or CPU 2 IO2 RTQPS RTQPSCPUQPSIOQPSIOQPSRT2 13. 312CPUQPS3RT 14. ----QPS-RTIO server serverserverPerf.jsp Iowait.jsp CPU 10 IO90CPUIOCPUIOQPSRT1 3linux aAB 4CPU b tomcatjdk 2 CPU cIO tomcatjdk 2CPU 2jspperf.jspiowait.jsp perf.war 15. Perf.jsp out.println("io time:"); out.println((t4 - t3)); out.println("ms
");%> 16. iowait.jsp 17. 1 perf.jspCPU 2CPUmodel name : Intel(R) Xeon(R) CPU5120 @ 1.86GHzstepping :6cpu MHz : 1862.650cache size : 4096 KBtomcatjvmJAVA_OPTS="-Xms256m -Xmx256m -Xss1024K -Djava.library.path=/root/jnicache/cachemap/cachemap-0.1/src"2iowait.jsp2CPUmodel name : Intel(R) Xeon(R) CPUE5320 @ 1.86GHzstepping: 11cpu MHz: 1861.916cache size : 64 KBtomcatjvmJAVA_OPTS="-Xms256m -Xmx256m -Xss1024K -Djava.library.path=/root/jnicache/cachemap/cachemap-0.1/src"CPU 4CPUmodel name : Intel(R) Xeon(TM) CPU 3.20GHzstepping : 10cpu MHz : 3192.783cache size : 2048 KBAB 18. - IOserver serverserver Perf.jsp Iowait.jspCPU 10 IO901ab ab -n10000 -c20 http://192.168.211.1:8080/perf/perf.jsp?cpuc=30000&ios=80 cpu time: 10 ms io time: 86 mstotal: 96ms AbQPS=174 RT = 114ms CPU 85% 20 2QPS1743001IO90ms45ms2CPU10ms5ms 19. -RTQPS IO serverserver server Perf.jspIowait.jspCPU10 IO90 CPU10IO451IO90ms45ms2abab -n10000 -c13http://192.168.211.1:8080/perf/perf.jsp?cpuc=30000&ios=40cpu time: 10 ms io time: 45mstotal: 55msAbQPS=176 RT = 74ms CPU 85%1350 20. -CPUQPS IO serverserver server Perf.jspIowait.jspCPU10 IO90CPU5IO901CPU10ms5ms2abab -n10000 -c40http://192.168.211.1:8080/perf/perf.jsp?cpuc=15000&ios=80cpu time: 5ms io time: 85 mstotal: 90msAbQPS=338 RT = 118ms CPU 85%40 21. -ABURL ab -n10000 -c10 http://192.168.211.1:8080/perf/perf.jsp?cpuc=?&ios=? AB URL QPSRT CPUCPU:10 ms perf.jsp?cpuc=30000&ios=80 2017511485%~IO:80 msCPU:5 msperf.jsp?cpuc=15000&ios=80 4033911885%~IO:80 msCPU:10 ms perf.jsp?cpuc=30000&ios=40 1317673 85%~IO40 msQPS 1RTIO IO 2QPSCPU 3hepser 22. hesperCPUIO200100ms1IO2QPQPS4549IOCPUIOCPUQPS1searchAuction.vmQPS90msQPS=*QPSQPS=1000ms/rtQPSCPU2QPS49190100msQPSCPU60%CPU 23. QPSwait322G2Gjvm 2G/256K=8000linux7409 -Xss)-Xss128K14349-Xss1M1896java.lang.OutOfMemoryError: unable to create new native thread at java.lang.Thread.start0(Native Method)500os1000LinuxTomcat 500 4urlqpsab -n100000 -c850 http://192.168.211.1:8080/test.jsp?delay=1Requests per second: 12692.86 [#/sec] (mean)Time per request: 66.967 [ms] (mean)ab -n100000 -c50 http://192.168.211.1:8080/test.jsp?delay=1Requests per second: 13592.44 [#/sec] (mean)Time per request: 3.679 [ms] (mean) 24. CPUQPSIOCPU10ms + IO 40ms = 50msCPU5ms45ms50ms45msqps2022CPU5100%CPU1CPU10ms5ms QPS RT CPUQPS 520 50 10ms*5*20=1000 5*20=100 X=9.122 45 5ms*x*22=10009.1 *22 = 200.22IO40ms20ms QPSRT CPUQPS 5 2050 10ms*5*20=1000 5*20=100 X=3 33.330 10ms*x*33.3=1000 33.3*3=100 25. 1proxyIONIOnginx2CPUCPU 26. 1jprof2jprof.py3TimeFilter4visualVMjprofiler YourKit 27. RT1IO IO HTTPHTTP2CPU forest 28. QPS1CPU2CPU3CPU85%QPS 29. *5~6Old 30. jvm3youngOldyoungeden+2survivor EdenEdenminorGCsurvivor young -XX:NewRatio=n-XX:NewSize=nXmn Survivor-XX:SurvivorRatio http://www.oracle.com/technetwork/java/javase/tech/vmoptions-jsp-140102.htmlOld survivoroldOldminorGCPermPerm OOM -Xms 31. 1Eden String str = new String(helloWorld)2EdenminorGCyoung(eden+1survivor)3minorGCEdensurvivorEdensurvivorGCsurvivorsurvivor43survivorcopyoldsurvivorOldOldXX:MaxTenuringThreshold5Old 32. jvmmemory2.jsp jsp21 2 33. 1eden > tomcatjvmJAVA_OPTS=-Xms256m -Xmx256m -Xss128k -XX:MaxTenuringThreshold=15Eden 15.812M S0 S1 1.938M Old 236.312MABab -n1000 -c1 "http://localhost/perf/memory2.jsp?m=15&s=10"15M10ms sleep11AEden15MA10ms15M210msAAEden15M3Eden0.812MminorGCEdenA15MsurvivorOld4Eden15.812MA15M5AEden0.812M1~5....Full GC 34. 22tomcatjvmJAVA_OPTS=-Xms256m -Xmx256m -Xss128k -XX:MaxTenuringThreshold=15ab -n1000 -c2 "http://localhost/perf/memory2.jsp?m=15&s=10"21AEden15MA10ms15M2B15MEden15.812MA15M0.812M3EdenminorGCA15MA15McopySurvivor4Survivor1.938MA15MA15MOld5Eden15.812M6B15MEden....Full GC Old 35. Eden30M(*) < Eden 1010MEden101M11Eden100MMinorGC100M10M~NMsurvivorsurvivorOld34 36. 321Mab -n1000 -c2 "http://localhost/perf/memory2.jsp?m=1&s=10"21AEden1MB1M8B1MEdenEden15M2minorGCB7A8A310ms8A4A1Mcopysurvivorsurvivor1.938M5minorGCsurvivorA1Mgc6Oldcopy....Full GC 37. 422Mab -n1000 -c2 "http://localhost/perf/memory2.jsp?m=2&s=10"21AEden2MB2M4B2MEdenEden14M2minorGCB3A4A310ms4A4A2Mcopysurvivorsurvivor1.938MOld52MOld....Full GC 38. 54survivor1.938M2.438M -XX:SurvivorRatio=6 tomcatjvmJAVA_OPTS=-Xms256m -Xmx256m -Xss128k -XX:MaxTenuringThreshold=15 -XX:SurvivorRatio=6ab -n1000 -c2 "http://localhost/perf/memory2.jsp?m=2&s=10"21AEden2MB2M4B2MEdenEden14M2minorGCB3A4A310ms4A4A2Mcopysurvivorsurvivor2.438M5minorGCsurvivorA2Mgc6Oldcopy....Full GCjvm 39. 1edenminorGC2survivorcopyold(survivoreden)3youngminorGC GCjvmGCyoung1/42gyoung500Msunsurvivoryoung1/8young500Msurvivor60M2M50Msurvivor 25OKjvmGCGC 40. 2Gyoung500Msurvivor60M=(survivor/) survivor=young/(6~8) young=(-Xmx) / 4Old6060/60=1M3060/30=2M12Old=(survivor/)=(survivor/) * gcjvm 41. 1minorGCedenminorGC2eden= Eden /QPS * minorGC()...minor gc(minor gc-minor gc)/(xminor gc * QPS)Detail GC.log2010-08-18T14:48:42.637+0800: 767442.589: [GC [PSYoungGen: 504824K->1526K(507904K)] 981442K->478201K(1671168K), 0.0231660 secs] [Times: user=0.01 sys=0.00, real=0.03 secs]2010-08-18T14:48:46.751+0800: 767446.704: [GC [PSYoungGen: 504644K->3330K(507072K)] 981319K->480109K(1670336K), 0.0120540 secs] [Times: user=0.01 sys=0.00, real=0.01 secs] 42. Gc.logDetail GC.log2010-08-18T14:48:42.637+0800: 767442.589: [GC [PSYoungGen: 504824K->1526K(507904K)] 981442K->478201K(1671168K), 0.0231660 secs] [Times: user=0.01 sys=0.00, real=0.03 secs]14:48:42young504824K1526K 503298K 981442K478201K503241K503298K - 503241K = 57KOld 57K2010-08-18T14:48:47.551+0800: 767446.704: [GC [PSYoungGen: 504644K->3330K(507072K)] 981319K->480109K(1670336K), 0.0120540 secs] [Times: user=0.01 sys=0.00, real=0.01 secs]14:48:47young504644K3330K501314K 981319K480109K501210K501314K - 501210K = 104KOld 104KGC5s5s 104K old 1Old 60/5 * 104 * 60 /1024= 73.125M 43. 132jvm cache400MOld1G600M 1NIOmmaposjvm 290%cache10% forest 3jvmOS2()null1 request.setAttribute("a",new XXX()); webMap2 // 1. Object xxx = new XXX(); // CPU XXXMethod.invoke 2. Object yyy = method.invoke();//10ms 3.zzz = yyy + xxx; 4.xxx=null;Method.invoke XXX // CPU 1. Object yyy = method.invoke();//10msxxx xxx // 2. Object xxx = new XXX(); minorGC 3. zzz = yyy + xxx; 4. xxx= null;34100 44. 1QPSRT2QPSRTQPS(CPU)IO = / * 3=((+cpu)/cpu) * cpu45IOIOIOproxywebww6IOIONIO71010 = (survivor/)(detail5) = (survivor/) * 85~6= Eden /(QPS * minorGC()) 45. Time for Q&A 46. Detail-QPSGC 1 2 CPU H TPS RT(s) Load FGCJVMApacheJboss Util% Detail300 1 ajp=200 69.95 0.17 66.15%4.97 1 N.A. 15 2031 Detail30018 ajp=200 55.35 1.80 74.60%7.10 31.9 JVM1Detail 100FGC Detail3001 ajp=1566.45 0.17 63.51%4.64 1 N.A.15 Detail300 15 ajp=1569.86 1.43 71.11%5.53 1 JVM2 15FGC 100 47. 1=((+cpu)/cpu) * cpu2(GC/rt)*( *thm)