ActiveRain Technology Update: May 13

Everyone,

I hope everyone had a GREAT week and has amazing plans for the weekend.  I have actually had a great week.  I have made some amazing new friends, spoke to amazing people (Greg, Donna, Craig, David, Karen, Tammy, Katerina, Mimi, Maureen, Todd, and many others), AND I am more energized by ActiveRain than ever before.  It's a beautiful day in Seattle.  It feels like summer is around the corner, which is a big deal in Seattle.

As I mentioned in my blog post last Friday, improving the stability of our technology and communication about our technology here at ActiveRain are jobs 1, 2, & 3 for me.  I am committed to talking to everyone here on ActiveRain regularly about our technology, even when there may not be anything earth shattering to say.

We are making great progress in building a stable foundation for ActiveRain and improving its performance.  As I mentioned before, there is no silver bullet and we may experience further outages in the future.   We are taking it one step at a time.  We are learning what we need to do.  We are investing in the right resources to make sure get to the root of the problem and solve it.

Here's this week's progress (you may want to grab a cup of coffee, tea, beer or beverage of your choice):

Our Team - Where There Was One, Now There Are Six: 

• To-date, my greatest failing as CEO of ActiveRain was not notifying everyone upfront that our CTO had resigned and walked out the door on the day I started at ActiveRain (actually within one hour of my walking in the door).  He had come to ActiveRain to work with Jon.  I am not Jon.  I spent three weeks in vain trying to convince him to come back and give us a reasonable transition.  I failed.  At the point I knew in my heart that we would not come to terms with him (probably end of last week), I should have openly told the community.  I am sorry I did not and have learned an important lesson about ActiveRain. 
• I apologize.  I own the drama of the week.  Period.
• We have assembled a great team that is tackling the core technology challenges underlying ActiveRain. 
• Chris Paredes is our technology lead here at ActiveRain.  He started several weeks before I did.  Chris has a great background in the technology that ActiveRain was built on.  Chris is committed and doing a stellar job, learning ActiveRain and troubleshooting problems as they arise.  Here is a pic of Chris and his dog Elise here in the office.   Here is a link to Chris's blog so you can learn more about him.
• Alex Lange is stepping in to play a more hands-on role to help us with the technology at ActiveRain.  Alex is the current CTO at Market Leader and is the former CTO at Roost and Chief Architect at Art.com   I have worked in and with many technology companies and Alex is probably the most capable CTO I have ever seen.  He delivers.  He is realistic.  His is on or ahead of time.  It is great to have him in our corner.
• Chris Beahm is the Production Operations Manager and Patrick Lesher is the  Sr. Linux Systems Administrator at Market Leader.  With our systems and database issues, Chris and Patrick are digging in to figure them out and help get the site back up.  Just to be clear, in the tech world, there are lots of different kinds of specialists.  In technology, there are user interface and front-end developers, there are middleware guys (how does the user interface talk to the database and the infrastructure?), there are database specialists (how does data get stored, how do we access it?), and there are operations and infrastructure specialists like Chris and Patrick (and Alex Lange) who design and maintain your infrastructure (boxes, cables, bandwidth and other hardware) to have optimal performance.  Prior to four weeks ago, ActiveRain had one talented developer straddling across many different technology specialities. 
• We have hired one consultant who is a rockstar Ruby on Rails developer who actually worked at ActiveRain from 2008-2009.  He is awesome.  Smart, fast, humble.
• We have also hired an incredible technology infrastructure consultant that is going through all the technology plumbing here at ActiveRain to help us create the plan to improve the technology.  We are learning that there are some basic things that can be updated to improve the performance and reliability of ActiveRain.  
• Once we have all the information and the plan in place, we will communicate that plan as much as we can.

Communication with the Community When ActiveRain is Down

• One of my greatest frustrations coming into ActiveRain is that there was no defined communication plan when the site is down.  Communication is critical and we have heard that from the community.
• I personally don't like hearing that things have gone "pear shaped" or that I can't "find my keys."
• We want our community to build their businesses and lives on ActiveRain.  
• On Monday, we had a meeting and defined how we are going to communicate with our members when the site is not functioning.
• Here is the current plan (we are open to feedback):
  1. We will put up a page on ActiveRain notifying you of the status of ActiveRain (so you don't need to go somewhere else to understand where we are). You can see a sample version here.
  2. We will continue to communicate on FaceBook and Twitter.
  3. We encourage you to contact us if you have any concerns.  There is a contact form on the maintenance page as well people can contact us by FaceBook, Twitter and phone.
  4. We will send out an email to the base if we believe there is an unusual circumstance where downtime may be significant (I am not sure what the time period is right now to be honest.  We are figuring this out.)
  5. I will post after an outage on my blog in terms of what happened and what we are doing about it.

Relationship with Rackspace

• We currently use Rackspace as our "managed hosting provider."  In other words, our 8 servers sit in a temperature controlled building in Chicago owned and operated by Rackspace.
• We pay a premium for their "fanatical support" - their tag line.
• I notified Rackspace that we are not happy with the level of support and service we are getting from them.  I have been on the job for 4 weeks now and the site has been down several times since I started and this is not acceptable.  We will look for other solutions if they are not able to solve this problem.  
• They are now scrambling to gather data and put together a plan to give us better support and stability.
• We should have a phone call in the near term to discuss how our relationship with Rackspace continues in the future.

What We Have Learned This Week

• We have made great progress this week.
• I talked with Alex Lange and our technology team about what we have learned from a deep dive into the "plumbing of ActiveRain."
• In general, my take-aways from the discussion are that the settings of our hardware at Rackspace need to be evaluated (as the traffic and requirements on the site have grown/changed), our database needs a regular vacuuming and maintenance (as I mentioned last week. Technical peeps: click here for more information about vacuuming a PostgreSQL database), and we are setting up the right monitoring to further understand the underlying issues.

Going Forward Plan:

• We will be implementing changes over the next week to not only stabilize, but improve performance.
• Over the next month, we'll continue to find the issues that have plagued our performance.   You should be seeing ever increasing improvements very soon.
• We plan on implementing a new architecture that can scale for years to come.  Part of this will include separating the community technology from the internal systems.
• Once we finish these items, we'll start improving ActiveRain and add new features.

Just to clarify, our jobs 1, 2, and 3 are stabilizing and improving the performance of ActiveRain.  Once we are confident in the foundation, we can start addressing bugs and building new features.  I want our team laser focused on the issue of stability and performance.

I reassure you that ActiveRain is NOT going anywhere and it is getting better everyday.  We are learning and our approach to the technology will bear fruit.  I have seen a number of members looking at WordPress as a complement to their ActiveRain blog.  We have ALWAYS enabled you to move your content from ActiveRain to WordPress if you want a back-up or feel like you want more control over your content.  ActiveRain and WordPress are complements AND I believe ActiveRain exists for a completely different reason than WordPress.  If you don't believe me, check out my collage competition in terms of what ActiveRain means to our members.  I wonder what a collage would look like if WordPress asked its members what WordPress means to them. 

ActiveRain has always been about the people, the connections and the community.  We need a strong foundation to build a house, a village and then a city.

As always, I look forward to hearing from you.  Call me or email me - 206 470 2908.  I pick up the phone.  I am also on our sales and customer service phone queues, so don't be surprised if you call in at some point and talk to me, like Fred Ehmann today.

Niki