http://www.bobsgear.com/dashboard.action?maxRecentlyUpdatedPageCount=40  
Recently Updated

.

Finally found the memory leak problem with my wikipedia parsing. It was a quirk of the LINQ database stuff keeping things in memory. The program is now happily processing 100 articles every 0.4 seconds, which is amazingly better than the 10 seconds it was taking to process 100 articles yesterday morning. Occassionally it speeds up and processes 100 articles in only 0.1 seconds.... And not having all the extra memory sitting around being held means the program is not gradually slowing down either.

How I structured the program:

• There is now one thread parsing the XML file, sending its results to a background thread
• The backgroudn thread does some extra parsing, and then background thread queues the objects ready for the database insert
• two database insert threads grab 250 articles at a time from the queue and batch insert them into the SQL Express database.

The XML parsing thread occassionally sleeps because it is much faster than the database threads, I dont want more than about 2500 articles sitting in memory waiting to be dumped into the database....

With the program current speed, it might take maybe 400 minutes to process entire wikipedia dump, down from maybe 6 days for the version of the program yesterday morning.

Some lessons learned:

• It is cheap to dispose of DataContexts with LINQ and create new datacontexts for each batch of database access. This gets rid of memory hanging around. This might not be the case for connecting with a remote database, the time to establish a new SQL Server connection to a remote server might be significant, this needs to be tested. But it is possible that multiple database threads could make up for this.
• Creating Regex objects takes a lot of time. Once a regular expression Regex is proven, move it outside of loops to be a global variable with the RegexOptions Compiled set. This alone resulted in a 50% speedup of my program.
• Despite using four threads, this program still only uses 12%-20% of an I7 processor. But now the disk drive light is on solid from all the database inserts. Occassionally the CPU usage jumps to 36% for short bursts, and the program seems to process records faster. Not sure why can't get the speed consistently up. More analysis could be done on the XML file parsing thread, but extra analysis results would increase the amount of stuff needing to be sent to the database, so it is a tradeoff.
• Having background threads for long running processing allows the UI to stay responsive.

A couple of useful articles:

• LINQ - My various notes on linq, such as Answers to Miscellaneous questions about LINQ, Troubleshooting memory leaks in LINQ
  • My experience 6/26/2010: Noticed in WikipediaToConfluence application: Written with background threads creating their own SQLMetal generated DataContext, and recreating a new DataContext every hundred records or so. Even without ever increasing memory usage, the Windows Vista system eventually gets very slow, and appears to run out of resources. Things like menus can not be selected in Visual Studio, other things requiring memory allocations fail, eventhough the system says it has 1-2GB of free ram after extended runtime of the application.
• http://stackoverflow.com/questions/123057/how-do-i-avoid-a-memory-leak-with-linq-to-sql - DataContext should have short lifetime, and be recreated often.
• Error handling for primary index violations, learn how to rollback for SubmitChanges failure.
  • http://social.msdn.microsoft.com/forums/en-US/linqprojectgeneral/thread/b80b1361-977b-4443-87ef-a856feba0c84/

This concerns revision 484 of [wminfra:WikipediaToConfluence] project.

Note: A PC can still become quite unusable over time if a program is running and doing a lot of updates to a database on the same machine. This can resemble a memory leak, but it doesnt show up in task manager. The machine may be almost unusable, unable to popup menus, programs may have strange errors, etc. Stopping and starting, restarting, the SQL Server service will restore the machine to proper operation. If SQL Server is not limited in the amount of memory it can use, it will try to use all the memory in the machine, but it uses it in a way that doesn't show up in task manager.

The other day I was tracking my diet for a couple of days, and discovered some big deficits of certain vitamins.

A friend of mine is a wellness doctor. I asked him for his opinion on vitamin supplements. Does he think vitamin supplements are useful, or are they a bunch of hype?

He said he feels they are very useful. The vitamins we have the most problem not getting enough of are:

• Vitamin A
• Vitamin D
• Vitamin E
• Vitamin C

I asked him what he thinks about the U.S. RDA numbers for vitamins. I've heard that they are set just above the minimum to keep you from getting diseases like scurvy, but not high enough to give real health. He agreed.

What about vitamin B-12? My doctor friend said that is very important for vegetarians and vegans to supplement.

I told him I had some trouble finding a vitamin supplement without extra iron. He asked if I cook with an iron skillet. Cooking with an iron skillet is a great way to end up with extra iron in the diet!

He takes 1000 mg. a day of Vitamin C, 1000 mg. of vitamin D. He said we have a problem with having enough vitamin D because we don't get enough sunlight.

My doctor friend said he had learned about the importance of vitamin supplements from watching a friend of ours, Dr. Merton Shelton who lived into his 90's and was a big believer in taking vitamin supplements.

Related pages:

• Problems with vitamins

Hmmm... Almost a year since my last blog posting in this space. That's the problem with having too many places to post things at.....

What have I been doing in the last year?

• Learning how to make beads and glasswork, lampworking, and when they don't turn out right, melting them down into marbles!
• Doing further development on the Hivewiki software. It's now running on another server also, with a more up to do date version, Webmill.us.
• Reading books:
  • Napoleon Hill's Think and Grow Rich - Working on cliff notes and a wiki version of it
  • Journey Round My Skull - Story of survivor of one of the earliest open brain surgeries
  • Do Cats Think? - Delightful book that answers "Of course they do!"
  • Health Power - Health By Choice, Not Chance

Think I've finished some others recently too, have to come back here and update the list.

In a search on Amazon using AWS or the Amazon website itself, for

    web site content

Link #13 is:

The Longing (2003) Plot Synopsis: Cyrus, Tim and Erik were high school buddies and meet again ten years later. They spend the summer together in Brazil. All of them are occupied in dealing with experiences from their past but new developments make them face their inner conflicts and their friendship with each other. Plot Keywords: Male Frontal Nud?ty | Homos??ual | Murder | Nud??y | Redemption | S?x | G?y | G?y Interest | G?y S?x | Cybers?x | Search For Parent | S?x On The Beach

What in the blazes does this have to do with website content? How did this happen?

I am very disgusted that AWS would return such an obviously out of place piece of garbage in the middle of a site about web design! What do I need to do to filter such trash out of the results I am getting from Amazon? I am trying to run sites that are family and child friendly, not sites that push this kind of stuff in people's faces when they aren't looking for it.

Even more stupid, this garbage outranks Net nanny. Here is the AWS descriptions in order for this search:

The Longing Jürgen Brüning $19.99(USD) -------------------------------------------------------------------------------- Ready for an admittedly tawdry but s?xy melodramatic romp through sun-drenched Brazil with two h0t men? Does Saved by the Bell s A.C. and Zack locking l??? and l???? on a white-sanded beach turn you on? (Editor's note: Yes I obscured a couple of words. The actual description is about twice as long and much more graphic.) -------------------------------------------------------------------------------- Net Nanny 5 Web Site Filtering Blocks Adult Content $39.95(USD) -------------------------------------------------------------------------------- Filters allow you to block objectionable material on the Web, while enabling access to everything else. For parents, this means you can reliably manage what Web sites your children see. Kids need to be able to access much of the Web for its informational, educational and entertainment content. However, a lot of sites out there are inappropriate and potentially damaging for children, featuring illicit content or promoting hate and violence. Get a monthly updated block list of inappropriate Web sites. Know exactly what your kids can and cannot see. Fully view and edit block lists. Create a "whitelist" of certain Web sites so that only those sites, and no others, may be visited or viewed. Create keyword lists to filter out inappropriate, vulgar or sexual terms on Web pages, in chat rooms and in emails. Use pop-up blockers to keep your screen from filling up with advertisements from adult sites and other over-zealous advertisers.

Maybe someone has an idea how I can run AWS search results through Net Nanny to keep my sites from displaying this trash and getting banned.....

Today a telemarketer called me from "Doctors Group", (732)242-6502, trying to schedule me for a Chiropractic screening. After having fun with the telemarketer asking her what movie was he screening, I asked her where was the doctor located, since I didn't recognize the 732 area code. I learned a lot about geography today, including new states I didn't previously know about (like Cincinnati), cities like Petaluma being moved to those new states, and more?...

Here is an incredible story of a homeless fellow trying to go from living in a cave he created to living in an apartment.... Old habits die hard, it's impossible to take someone farther than they can see.... I can identify with parts of the story. I escaped California's sky high rental market for a few years by living in an RV. It was a fun time, nice to save $1000 or more a month. However, I still have a habit of scanning for good places to pull over for the night, or looking for electrical outlets in parking garages that I might plug into..... For someone suffering from mental illness, and drug addictions, they may have lost the higher brain functions that would allow them to override the animal like tendency to live according to instinctual responses.

Al-Qaida cited in persecution of 200 million Christians

A report by Britain's intelligence service MI6 reveals for the first time an estimated 200 million Christians in 60 countries are now facing persecution orchestrated, in part, by al-Qaida.

By Michael Rush Monday, June 4, 2007

The report has been sent to the pope and church leaders. It is the first time the secret intelligence service has shared its information with religious leaders.

"We do so because we believe the situation is extremely serious," said an MI6 source.

The report is based on details from MI6 agents stationed in the countries. North Korea is identified as the world's worst repressor of Christians. More than 50,000 Christians are incarcerated in work camps because they refuse to submit to the extreme views rigorously enforced by the country's dictator, Kim Jong-il.

An estimated 40,000 Christians are imprisoned in China for their beliefs. The MI6 analysis calculates there are some 70 million "active Christians" in the People's Republic, all living in a climate of fear because of their beliefs.

The woman in question, Ms Lina Joy, 43, was born Azlina Jailani to Malay parents and brought up a Muslim. In 1998, she converted to Christianity and changed her name.

But the National Registration Department here refused to reflect her change of religion in her identity card.

Here appeal went all the way to the Malaysian Federal Supreme Court 3 judge tribunal ruled along religious lines, 2 to 1 against her, saying that she must go to the Syariah Courts for this issue. More on Malaysia's Civil Federal Supreme Court Hands Over Non-Muslim to the Religious Syariah Courts.... Malaysian Lina Joy and the Malaysian Federal Supreme Court Decision and the issue of Jurisdiction...

I watched as our new tomcat Braveheart jumped up on the bathroom sink, and was staring very intently at the mirror image of his sisters eating at the food bowl in the hallway outside the bathroom. His own reflection was much closer to him, and bigger, but he ignored that, and was obviously very interested in watching his new sisters via the reflection in the mirror. Was this cat demonstrating that cats are self aware? I research this more and posted a survey of Mirror tests, self awareness, and cats...

Very interesting to see M from the ames Bond movies (Judi Dench) playing an old maid history teacher, with her dairy she is always writing in. "I'm just a battle axe, not popular, but they respect me" she says. And Cate Blanchett as a new art teacher named Shiva. The accents in the dialogue are sometimes hard to catch. "After lunch, a mortifying family tradition" is how M describes Cate and family dancing. The movie told as narrative from M's point of view. "She spoke of her vile mother, her grief over the death of her father, glory days when when Richard scooped her up, and then jetisoned the wife and children for what she called a nonstop f!fest. It's a peculiar trait of the privileged, immediate incautious intimacy. But Shiva went well beyond the tendencies of her class. She was utterly candid. A novice confessing to the mother superior." Movie - Notes on a Scandal...

Favorite quotes:
Heard at a dental conference from a presenter holding up a small vial of Novocaine anesthetic to the audience: "This is why you have a practice."