| In this set of pages on Investigating A Slow Confluence Installation - Version 3.3.1, I document the steps I took to try and identify slow running operations on my copy of Confluence 3.3.1. I recently added about 10K pages and labels for an EGW topical index, roughly doubling the number of pages (not not total size of content). This should account for the recent slowdowns and non-responsive Confluence that I have seen.
In this set of topics I discuss how I examined the database, stack dumps, dealt with Confluence hangs, logged bugs with the vendor, examined robot activity, and took other defensive measures to keep my copy of Confluence running. Please let me know if these topics helped you run your copy of Conflunece. You can use the "Add comment" on any of these pages to let me know. Please include your email addres in your comment, I will keep a copy of your email so we can correspond about Confluence, but quickly remove it from the comment you place here. |
One of the cooler tools that Atlassian has built in the last four years is the Hercules log analysis tool. I was always amazed four years ago when I would make a support request, and the technician woudl give me answers for the problem I was reporting, as well as make me aware of other issues I didn't even know were a problem. Now Atlassian has a tool called Hercules that analyzes Confluence logs automatically.
My support request about the rogue CPU hogging thread in Confluence was met with a response by a technician claiming out of memory issues were the cause of my problems. He noted several other things from the Hercules logs also. There was no response about the specific thread stack trace I was asking about. They were also happy I disabled the global usage plugin, which is known not to scale well. (I get a max of 10K hits a day, that doesn't seem like a lot to me, but tech support reflexively recommends getting rid of that plugin. So I already disabled it, and am still seeing performance problems.) So I logged a response, with some more logs from yesterdays 3:00pm restart, which are lacking in any out of memory errors.
I decided to use the Support Tools plugin again to let Hercules analyze my current logs since yesterdays restart. Here is it's output for my logs, I switch to richtext editor so that all the urls are saved when I cut and paste it (it would be nice if Hercules would include the excerpt from my logs for each case):
Hercules found 66 matches for known problems ( show all / show top 10 ) in the following log file: /home/confluence.bobsgear/data/logs/atlassian-confluence.log
- Plugin throws java.lang.UnsupportedClassVersionError Unsupported major.minor version 49.0 due to JDK
3 matches, last match on line 10809 - Unable to Upgrade on SQL Server due to 'Foreign key has implicit reference to object which does not have a primary key defined on it'
1 match on line 10805 - Error appears when attempting to import Microsoft Word 2003 formatted documents
1 match on line 10803 - Rt2 - Incompatible Magic Value
1 match on line 10799 - Error using Doc Import in News.
1 match on line 10797 - Problem installing on Oracle schema when other schemas in the same database have one or more tables sharing the same name as any of Confluence's tables
3 matches, last match on line 10776 - Performance and Stability Issues with Office Connector 1.6 or 1.7
2 matches, last match on line 10751 - Unable to Install Oracle on Websphere due to Classloader Preferences
1 match on line 10747 - Confluence Rebuild Indexing task fails to complete.
1 match on line 10728 - Unable to Export to HTML due to Page Title with * Symbol
1 match on line 10648 - MailException Domain Name Required for Sender Address
1 match on line 10646 - Dynamic TaskList Macro Not Working due to Scaffolding Plugin
1 match on line 10644 - CSRF attack message thrown when JSESSIONID is changed
1 match on line 10642 - Unable to Configure Confluence to Run as a Service on Tomcat
1 match on line 10640 - Confluence Does Not Start and Throws Spring Context Error in Linux
1 match on line 10630 - Unable to Install Plugins Using MS SQL Due to Transaction Log Exceptions
2 matches, last match on line 10621 - "Create Personal Space" menu item missing for some users, particularly LDAP users
1 match on line 10617 - Poor Performance from Excessive Garbage Collection in the Excel Plugin
1 match on line 10603 - Slow Page Rendering of Large Pages due to HTTP POST Limitations
3 matches, last match on line 10522 - Documentation Theme Fails to Display the Navigation Bar in Some Spaces
4 matches, last match on line 10520 - Unable to Export Space due to NullPointerException
2 matches, last match on line 10518 - The URL for this File is too Long for the Application due to Path Limitation
1 match on line 10516 - NullPointerException when Sending a Test Email
1 match on line 10510 - Use jTDS Driver 1.0.3 or Later for JDBC Connection
2 matches, last match on line 10506 - Error appears when attempting to import, display or edit Microsoft Office 2007 documents
3 matches, last match on line 10504 - Confluence ships with a javax.xml.namespace.QName class
1 match on line 10500 - Connection readOnly Mode is not Enforcable After the Connection Has Been Established
1 match on line 10498 - Gallery Macro is not able to sort by comments when there are combination of empty and non-empty comments
1 match on line 10496 - Getting Error Rendering Template for Decorator Root Error in a Page after Upgrade
1 match on line 10494 - PageNotFound action can render Confluence inoperable
1 match on line 10492 - System Error when Configuring IM Presence Plugin
1 match on line 10488 - Upgrade to Confluence 3.0 with MySQL database fails with messages like "specified key was too long"
1 match on line 10399 - MalformedInputException when rebuilding the Did You Mean search index
1 match on line 10391 - Hibernate caching in a cluster causes performance issues
1 match on line 10385 - Webdav Launcher fail to install in Firefox with error number 207
1 match on line 10381 - The Datetime Zone Id is Not Recognised
1 match on line 10360 - Unable to Setup Microsoft Network Drive to Connect to Webdav
1 match on line 10354 - Older version of Page Tree plugin not removed during upgrade to 2.10.x
1 match on line 10248 - XML Backups Cause Performance Problems or Out Of Memory Errors
1 match on line 10246 - Plugin being disabled after being upgraded or re-enabled
1 match on line 10238 - This Operation Breaks == WARN Messages
1 match on line 10232 - LDAP Error Code 49
1 match on line 10228 - Truncated Confluence.cfg.xml File due to Insufficient Disk Space
1 match on line 10224 - Unable to Start Node due to no Cluster License
1 match on line 10200 - MySQL Table is Marked as Crashed and Should be Repaired
1 match on line 10198 - Confluence Returns A Blank Page When Visiting A Space
1 match on line 10194 - Unable to Start Confluence after LDAP Integration
1 match on line 10192 - NoClassDefFoundError in com.atlassian.confluence.core.actions.NotFoundAction
1 match on line 10190 - Officially support Weblogic 10.x
1 match on line 10044 - Non-Unique Space error when importing SnipSnap data
1 match on line 9776 - Confluence Takes too Long to Startup
1 match on line 9759 - Problem creating RSS Feed with Feedbuilder due to IBM JDK
1 match on line 9457 - Fatal JVM Crash - 'java.lang.OutOfMemoryError .. Out of Swap Space?'
1 match on line 9455 - ClassCastException gnu.mail.handler.TextPlain when Sending Test Mail due to GNU Java
1 match on line 9232 - MySQL has Performance Problems when Pages Have Many Revisions
1 match on line 9226 - After upgrade, editing via WebDAV fails
1 match on line 9224 - Using square brackets around a long block of text causes error page: "Data too long for column 'DESTSPACEKEY'"
1 match on line 9222 - Can't Create or Write to File Error
1 match on line 9203 - Cannot upgrade to Confluence 3.1.x when using Crowd integration
1 match on line 9127 - Confluence Crashes due to java.lang.OutOfMemoryError PermGen Space Error
1 match on line 9125 - Invalid Email addresses cause problems with sending mail and indexing
1 match on line 8694 - CLONE -System error when adding users to a group if the group name contains certain characters
1 match on line 8692 - Encountered _APP_UNKNOWN error when Using JIRA Issues Macro
1 match on line 8686 - Recently Updated Macro Does Not Display More than 200 Entries
3 matches, last match on line 7424 - MissingResourceException: Can't find resource for bundle com.atlassian.confluence.util.i18n.CombinedResourceBundle
11 matches, last match on line 6598 - Confluence performance / installation issue when using Java 6 Update 18 64 bit
1 match on line 74
I made a copy of these logs
cp atlassian-confluence.log ../../logscopy/atlassian-confluence.herc2010-09-13.log cp ../../current/logs/catalina.out ../../logscopy/catalina.herc2010-09-13.out
I grabbed the catalina.out since maybe it's thread dumps will give some clues what was happening at the time of the above log entries.
I'll look at these in more detail later. The ones worrying about Oracle are almost definitely not applicable to my instance.