 | In this set of pages on Investigating A Slow Confluence Installation - Version 3.3.1, I document the steps I took to try and identify slow running operations on my copy of Confluence 3.3.1. I recently added about 10K pages and labels for an EGW topical index, roughly doubling the number of pages (not not total size of content). This should account for the recent slowdowns and non-responsive Confluence that I have seen.
In this set of topics I discuss how I examined the database, stack dumps, dealt with Confluence hangs, logged bugs with the vendor, examined robot activity, and took other defensive measures to keep my copy of Confluence running. Please let me know if these topics helped you run your copy of Conflunece. You can use the "Add comment" on any of these pages to let me know. Please include your email addres in your comment, I will keep a copy of your email so we can correspond about Confluence, but quickly remove it from the comment you place here. |
As part of Investigating A Slow Confluence Installation - Version 3.3.1, I found that Confluence finally just was giving proxy errors to every request, so I killed it with a -8. I ran several kill -3 to get some threaddumps into the logs. Then after killing confluence, moved all the logs to a bug report directory.
While confluence was down, here is a sample of the http_access log:
84.93.169.215 - - [10/Sep/2010:19:12:32 -0700] "GET /s/1923/3/1.0.8/_/download/batch/com.atlassian.confluence.plugins.drag-and-drop:drag-and-drop-for-view-content/com.atlassian.confluence.plugins.drag-and-drop:drag-and-drop-for-view-content.js HTTP/1.1" 200 1782 "http://webcache.googleusercontent.com/search?q=cache:qUkf8j6eljUJ:www.bobsgear.com/display/ts/Premiere%2BPro%2BCS4%2BNot%2Bpicking%2Bup%2Baudio%2Bfrom%2Ban%2BMPEG%2Bfile+adobe+conforming+no+audio&cd=8&hl=en&ct=clnk&gl=uk" "Mozilla/5.0 (Windows; U; Windows NT 6.0; en-GB; rv:1.9.2.9) Gecko/20100824 AskTbUT2V5/3.8.0.12304 Firefox/3.6.9 (.NET CLR 3.5.30729)" 65.55.3.201 - - [10/Sep/2010:19:14:24 -0700] "GET /display/cndopen/Google+Using+Human+Evaluation HTTP/1.1" 502 550 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:14:54 -0700] "GET /display/cndopen/Google+Using+Human+Evaluation HTTP/1.0" 502 550 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.97 - - [10/Sep/2010:19:14:57 -0700] "GET /display/AUM/Converting+Evils+in+Personal+Astrology+into+Positive+Experiences HTTP/1.0" 502 612 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 75.198.141.84 - - [10/Sep/2010:19:15:05 -0700] "GET /favicon.ico HTTP/1.1" 502 482 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 5.1; MindQuizSearchToolbar 1.1; GTB6.5; Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1) ; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.4506.2152; .NET CLR 3.5.30729; InfoPath.2; AskTB5.6)" 207.46.199.45 - - [10/Sep/2010:19:15:13 -0700] "GET /display/pmbooks/Scanned+Books HTTP/1.0" 502 518 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 178.154.162.29 - - [10/Sep/2010:19:15:14 -0700] "GET /label/girls+them+vote HTTP/1.1" 502 502 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 173.19.95.165 - - [10/Sep/2010:19:15:33 -0700] "GET /favicon.ico HTTP/1.1" 502 482 "-" "Mozilla/4.0 (compatible; MSIE 7.0; Windows NT 6.0; Win64; x64; GTB6.5; .NET CLR 2.0.50727; SLCC1; Media Center PC 5.1; .NET CLR 3.5.30729; .NET CLR 3.0.30618)" 76.102.190.149 - - [10/Sep/2010:19:15:56 -0700] "POST /pages/doeditpage.action?pageId=21397568 HTTP/1.1" 502 507 "http://www.bobsgear.com/pages/editpage.action?pageId=21397568" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.2.8) Gecko/20100722 Firefox/3.6.8" 207.46.12.236 - - [10/Sep/2010:19:16:07 -0700] "GET /display/any/Bay+Area+Housing+Bubble+Wiki HTTP/1.1" 502 540 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 207.46.12.236 - - [10/Sep/2010:19:16:07 -0700] "GET /display/~admin/2007/06/03/Lina+Joy+Loses+Appeal%2C+Must+Endure+Oppression+of+Malaysian+Syariah+Courts?showComments=true HTTP/1.0" 502 658 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:16:24 -0700] "GET /display/egw/Discipline HTTP/1.1" 502 504 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.96 - - [10/Sep/2010:19:16:35 -0700] "GET /display/ibs/Grad+School+The+Snooze+Button+On+The+Alarm+Clock+Of+Life HTTP/1.0" 502 596 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:16:54 -0700] "GET /display/egw/Discipline HTTP/1.0" 502 504 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.199.32 - - [10/Sep/2010:19:17:28 -0700] "GET /display/~admin/Prius+Battery+Replacement+Problems+Wiki HTTP/1.1" 502 568 "-" "Mozilla/4.0 (compatible; MSIE 6.0; Windows NT 5.1; SV1; .NET CLR 1.1.4322; .NET CLR 2.0.50727; .NET CLR 3.0.04506.648)" 207.46.199.38 - - [10/Sep/2010:19:17:50 -0700] "GET /display/ibs/When+Religion+Ruled+The+World+They+Called+It+The+Dark+Ages HTTP/1.1" 502 600 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 66.249.68.77 - - [10/Sep/2010:19:16:58 -0700] "GET /display/egw/Experience?showComments=false HTTP/1.1" 502 504 "-" "Mozilla/5.0 (compatible; Googlebot/2.1; +http://www.google.com/bot.html)" 178.154.162.29 - - [10/Sep/2010:19:17:21 -0700] "GET /label/like+me+why HTTP/1.1" 502 494 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 178.154.162.29 - - [10/Sep/2010:19:19:01 -0700] "GET /label/even+food+melamine HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 65.55.3.201 - - [10/Sep/2010:19:18:53 -0700] "GET /display/cndopen/Blogging HTTP/1.0" 502 508 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 178.154.162.29 - - [10/Sep/2010:19:19:03 -0700] "GET /label/even+food+melamine HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 65.55.3.201 - - [10/Sep/2010:19:19:03 -0700] "GET /display/any/Relation+of+Exercise+Growth%2C+Loading%2C+and+Active+Recovery+Phases+to+Growth+Hormone+and+Testosterone+Production HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:04 -0700] "GET /dashboard.action HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.89 - - [10/Sep/2010:19:19:05 -0700] "GET /display/ibs/Hamilton+Tcats HTTP/1.0" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:18:24 -0700] "GET /display/cndopen/Blogging HTTP/1.1" 502 508 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:06 -0700] "GET /display/egw/Apples+of+Sodom HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:07 -0700] "GET /display/ds/Adding+a+comment+to+a+page HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:08 -0700] "GET /display/any/dhea?showComments=true&showCommentArea=true HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 71.131.201.54 - - [10/Sep/2010:19:18:29 -0700] "GET /display/stocks/2009-11-20+XLT+Stock+Trading HTTP/1.1" 502 546 "http://www.google.com/url?sa=t&source=web&cd=1&ved=0CBIQFjAA&url=http%3A%2F%2Fwww.bobsgear.com%2Fdisplay%2Fstocks%2F2009-11-20%2BXLT%2BStock%2BTrading&rct=j&q=futures%20drop-base-rally%20indicator&ei=V0OITOCeIYbEsAOmqqnACg&usg=AFQjCNEDdHE1WLme4DbWIdzZ3uFZFL9HwA" "Mozilla/5.0 (Windows; U; Windows NT 6.1; en-US; rv:1.9.1.6) Gecko/20091201 Firefox/3.5.6" 65.55.3.201 - - [10/Sep/2010:19:19:09 -0700] "GET /display/egw/Book+illustration HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:11 -0700] "GET /display/cndopen/Website+Directories HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:12 -0700] "GET /display/bobsgear/Confluence+Example+Plugin+-+Confluencerpc HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:13 -0700] "GET /display/cndopen/Business+Cards HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:14 -0700] "GET /display/any/Who+ever+heard+of+an+Indiana+Trademark HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:16 -0700] "GET /display/any/Article+-+Morality+the+case+for+polygamy HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:17 -0700] "GET /display/any/Comparison+of+Presidential+Candidate+Positions?focusedCommentId=2064801 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 178.154.162.29 - - [10/Sep/2010:19:19:18 -0700] "GET /label/even+food+melamine HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 65.55.3.201 - - [10/Sep/2010:19:19:18 -0700] "GET /display/any/2+Samuel+11-12 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:19 -0700] "GET /display/egw/Book+work HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:20 -0700] "GET /display/AUM/Breakthrough+Options+for+Management+Science HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:22 -0700] "GET /display/egw/Affinity HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:23 -0700] "GET /display/egw/Church+work HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:24 -0700] "GET /display/egw/College+leader%2C+College+leaders HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:25 -0700] "GET /display/ibs/A+Friend+Is+Someone+Who+Still+Likes+You+Even+After+They+Get+To+Know+You HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:27 -0700] "GET /label/adjective+christian HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:28 -0700] "GET /label/73+college+program HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.93 - - [10/Sep/2010:19:19:28 -0700] "GET /login.action?os_destination=%2Flabel%2Fibs%2Fmans HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:19:29 -0700] "GET /display/wsgmfoods6hr/Food+Inc?showChildren=false HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:30 -0700] "GET /display/ibs/Equal+Rights+Not+Special+Rights HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:32 -0700] "GET /label/AUM/metaphysics HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:33 -0700] "GET /label/73+boardinghouses+college HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:34 -0700] "GET /exportword?pageId=4521985 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:35 -0700] "GET /label/132+psalm HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:37 -0700] "GET /display/wsewriting2hr/Navigation HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:38 -0700] "GET /display/quotes/Saint+Basil+The+Great HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:39 -0700] "GET /display/ibs/Happy+Alone HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:40 -0700] "GET /display/ssnet/Sermon+-+Knowing+Evil+-+Genesis+3 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 71.61.241.111 - - [10/Sep/2010:19:18:12 -0700] "GET /display/stocks/Spider+Scan+Tool HTTP/1.1" 502 522 "http://www.google.com/url?sa=t&source=web&cd=7&sqi=2&ved=0CCsQFjAG&url=http%3A%2F%2Fwww.bobsgear.com%2Fdisplay%2Fstocks%2FSpider%2BScan%2BTool&rct=j&q=pre%20market%20scans%20for%20gap%20stocks&ei=ruSKTKHqM4a8lQewxMW5Cg&usg=AFQjCNEsixuWuiCDuxhg9g9DFJW0xtUjUQ" "Mozilla/4.0 (compatible; MSIE 8.0; Windows NT 5.1; Trident/4.0; GTB6.6; .NET CLR 2.0.50727; yie8)" 65.55.3.201 - - [10/Sep/2010:19:19:42 -0700] "GET /display/ptm/Spelling HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:43 -0700] "GET /display/ibs/Real+Women+Drive+Trucks HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:44 -0700] "GET /display/lawgear/ADVANTAGE+HEALTH+PARTNERS+LP+v.+ACCESS+IV%2C+Gary+Dangerfield%2C+Gwen+Porter+C+03-04496+JF+-+US+District+Court+N.D.+California HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:46 -0700] "GET /display/egw/Fleshpot%2C+Fleshpots HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:47 -0700] "GET /display/egw/Home HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.199.52 - - [10/Sep/2010:19:19:48 -0700] "GET /display/ts/NTVDM+encountered+a+hard+error+on+Windows+2003+W2K3+Server HTTP/1.0" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 178.154.162.29 - - [10/Sep/2010:19:19:48 -0700] "GET /label/even+food+melamine HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 65.55.3.201 - - [10/Sep/2010:19:19:48 -0700] "GET /display/garnet/My+reality+show+ideas+related+to+Deadliest+Catch?showChildren=false HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.195.232 - - [10/Sep/2010:19:19:49 -0700] "GET /display/telemarket/Handling+Home+Improvement+Telemarketers HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:19:49 -0700] "GET /display/bobsgear/Project+Planning HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:51 -0700] "GET /display/egw/Cheering+word HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:52 -0700] "GET /display/bobsgear/Amazon+API+Macros HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:53 -0700] "GET /display/any/Internet+Gambling HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:54 -0700] "GET /display/ibs/A+Friend+Is+Someone+Who+Still+Likes+You+Even+After+They+Get+To+Know+You HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:56 -0700] "GET /display/egw/Temperance+campaigns HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:57 -0700] "GET /display/ibs/I+Support+Gay+Marriage+But+Only+If+Both+Chicks+Are+Really+Hot HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:19:59 -0700] "GET /display/garnet/My+reality+show+ideas HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:00 -0700] "GET /display/d/death+tax HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:01 -0700] "GET /display/cndopen/Book+Publishing+-+Tools HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:02 -0700] "GET /display/any/Sugar HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:03 -0700] "GET /display/any/Does+excess+sugar+in+the+diet+contribute+to+type+2+diabetes HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:04 -0700] "GET /display/ds/_Blue HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:07 -0700] "GET /display/egw/Religious+bodies HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:09 -0700] "GET /display/egw/Heathen+ceremony%2C+Heathen+ceremonies HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:12 -0700] "GET /display/healthnews/Obesity HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:14 -0700] "GET /display/egw/Look HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:15 -0700] "GET /display/egw/Cheerful+spirit HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:16 -0700] "GET /display/egw/Debasing+practices+and+thoughts HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:18 -0700] "GET /display/cndopen/Legal+Support HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:19 -0700] "GET /display/d/a+lo+divino HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:20 -0700] "GET /display/ibs/Got+Punk HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:21 -0700] "GET /display/ibs/Ok+Run+Along+And+Die+Now HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:23 -0700] "GET /display/bobsgear/Label+Maps HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:24 -0700] "GET /display/cndopen/Duplicate+Content HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:25 -0700] "GET /display/garnet/Mailbag HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:26 -0700] "GET /display/bobsgear/Testing+page+rename+in+wikitext+and+comments?showComments=false HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:27 -0700] "GET /display/babeinbunker/Book+Lists?showChildren=false HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:29 -0700] "GET /display/egw/Dietetic+reform HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:30 -0700] "GET /display/egw/Swine HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:31 -0700] "GET /display/any/Subchapter+S+Corporation+Tax+Issues?showComments=true&showCommentArea=true HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:32 -0700] "GET /display/AUM/Home HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:33 -0700] "GET /display/ibs/I+Have+An+Attitude+And+I%27m+Not+Afraid+To+Use+It HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:38 -0700] "GET /display/babeinbunker/Home?showChildren=false HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:39 -0700] "GET /display/egw/Christian+life HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:40 -0700] "GET /display/egw/Fanciful+thing HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:41 -0700] "GET /display/egw/Forest%2C+Forests HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:43 -0700] "GET /display/any/How+To+Have+a+More+Social+Cat HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:44 -0700] "GET /display/egw/Medical+institution+superintendent%2C+Medical+institution+superintendents HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.50 - - [10/Sep/2010:19:20:45 -0700] "GET /display/ibs/Having+The+Right+To+Do+Something+Doesn%27t+Mean+It%27s+Right+To+Do+It HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 178.154.162.29 - - [10/Sep/2010:19:20:47 -0700] "GET /pages/tinyurl.action?urlIdentifier=RgAm HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)" 65.55.3.201 - - [10/Sep/2010:19:20:48 -0700] "GET /display/egw/Flattering+inducement HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:49 -0700] "GET /display/egw/Distinguished+men HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:50 -0700] "GET /display/healthnews/Pages+about+sleep+at+BobsgearDotCom HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:52 -0700] "GET /display/cndopen/Faxing HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:53 -0700] "GET /display/ibs/I+Need+Someone+Really+Bad+Are+You+Realy+Bad HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:54 -0700] "GET /display/egw/College+game%2C+College+games HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:55 -0700] "GET /display/Notes/another?showComments=true&showCommentArea=true HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.199.198 - - [10/Sep/2010:19:20:56 -0700] "GET /robots.txt HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 65.55.3.201 - - [10/Sep/2010:19:20:56 -0700] "GET /display/any/Article+-+PALESTINE+-+Privilege+of+Polygamy HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:58 -0700] "GET /dashboard.action HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:20:59 -0700] "GET /display/AUM/Home?focusedCommentId=1671241 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 65.55.3.201 - - [10/Sep/2010:19:21:00 -0700] "GET /display/Notes/another+page+21 HTTP/1.1" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)._" 207.46.13.141 - - [10/Sep/2010:19:21:07 -0700] "GET /display/bobsgear/Got+Wiki?showComments=true&showCommentArea=true HTTP/1.0" 503 404 "-" "msnbot/2.0b (+http://search.msn.com/msnbot.htm)" 178.154.162.29 - - [10/Sep/2010:19:21:46 -0700] "GET /pages/tinyurl.action?urlIdentifier=RgAm HTTP/1.1" 503 404 "-" "Mozilla/5.0 (compatible; YandexBot/3.0; +http://yandex.com/bots)"
44 hits in a minute at 19:20. all 44 of them from robots. 55 accesses in the previous minute, and 53 of them were robots. Sigh...
Restarted confluence. It took
- INFO: Server startup in 160522 ms
Things were a lot better after the restart:
top - 19:26:36 up 1 day, 5:06, 5 users, load average: 0.52, 1.53, 2.45
Tasks: 141 total, 1 running, 140 sleeping, 0 stopped, 0 zombie
Cpu(s): 0.0%us, 0.3%sy, 0.0%ni, 99.7%id, 0.0%wa, 0.0%hi, 0.0%si, 0.0%st
Mem: 2301952k total, 2215816k used, 86136k free, 51496k buffers
Swap: 1081336k total, 176k used, 1081160k free, 738552k cached
PID USER PR NI VIRT RES SHR S %CPU %MEM TIME+ COMMAND
7722 root 15 0 2352 1084 812 R 0.3 0.0 1:15.64 top
1 root 15 0 2088 672 576 S 0.0 0.0 0:11.01 init
2 root RT -5 0 0 0 S 0.0 0.0 0:00.37 migration/0
3 root 34 19 0 0 0 S 0.0 0.0 0:00.12 ksoftirqd/0
4 root RT -5 0 0 0 S 0.0 0.0 0:00.02 watchdog/0
5 root RT -5 0 0 0 S 0.0 0.0 0:00.26 migration/1
6 root 34 19 0 0 0 S 0.0 0.0 0:00.10 ksoftirqd/1
7 root RT -5 0 0 0 S 0.0 0.0 0:00.00 watchdog/1
8 root 10 -5 0 0 0 S 0.0 0.0 0:00.37 events/0
9 root 10 -5 0 0 0 S 0.0 0.0 0:00.09 events/1
10 root 10 -5 0 0 0 S 0.0 0.0 0:00.04 khelper
Host was showing only 4% - 8% usage, but msnbot had gone away too. Yandex took up the slack and started coming in for more content, but CPU usage in guest 1% for java process, and host 2% was much better.
Going to have to [study the thread dumps to see what was happening before hung Confluence had to be restarted].
Labels:
None
Add Comment