Last night we went through a scheduled downtime that did not go as smoothly as planned. On Tuesday morning we received notice of the downtime from our datacenter and we sent out notification emails that day in order to provide as much notice as we could to our users. The maintenance was completely out of our hands, which is why we weren’t able to do this on a weekend. The third party insisted critical work was being performed, so we were given no say in the matter. The work was related to power outages, and they completed it in approximately an hour as they had predicted.

An issue occured, however, when Wufoo went back up with the way subdomains are resolved. Approximately an hour and a half after we came back up, we were notified that some accounts were not functioning properly because fo the subdomain issues. We were able to resolve the issue immediately and the servers became fully functional around 6am PST.

It seems that single points of failure continue to plague us when we experience downtime. As we did the last time we had an outage, we have scheduled time to sit down with our third party providers and figure out what steps need to be taken to avoid potential issues in the future. Our thanks go out to you for your understanding and patience as we continue to grow. We know many of you trust important parts of your businesses with us and we don’t take this responsibility lightly.