Since Tuesday’s outage on Tempo, we’ve been working hard to not only prevent that from happening again, but to further improve our system’s response time and optimize our resource utilization. This kind of spring cleaning and infrastructure maintenance was long overdue and took a backseat while we were building out the most recent update.
I’m happy to report, and hopefully you’ve noticed, that we’ve managed to not only stabilize our runaway memory consumption, but to dramatically improve response time over all (go Unicorn!). Now that things have stabilized and in fact improved, we’ll be working to further cut down on memory consumption in our application code. It’s very easy to have memory bloat in Ruby code, because Ruby doesn’t really release memory back to the system, so we’re profiling our code to cut back where ever we can.
As an aside, there’s this great Engine Yard post about memory bloat in Rails apps with pointers for pinpointing problem spots and cutting back. However, I take issue the notion that “it’s not a memory leak, it’s bloat.” That’s a lot of garbage, not returning unused memory to the system is a leak, and to pretend otherwise is a laughable bit of delusion. I agree that it’s important to recognize this as a known flaw of the platform, but to start thinking of it as a feature is probably not a good idea.
Tempo was down this afternoon shortly after 4pm EST due to a nasty memory issue. It’s fixed for now and the service is back up, running nice and snappy, but we still need to implement a permanent fix. Keeping the service available to our customers at all times is exactly the point of web-based software, and we’re committed to providing you the best product possible.
We’ve been running on Nginx-Passenger for a few months now, and been trying to fend off a memory leak that just keeps coming back, and when we don’t catch it in time, like today, it can turn into a big mess. We are dropping pretty much everything to make sure we don’t end up in this position again. We will post an update here once we’re confident that we have a permanent solution in place.
Thanks for your patience, and our apologies if you were unable to access Tempo when you needed it this afternoon.
Over the weekend we rolled out a pretty extensive update to Tempo, our time-tracking service. Some of the changes have been confusing for our customers, so we’d like to take a quick stab at some of the common questions we’re getting and put them here on the blog.
I got an email about my URL changing, is this real?
Yes, it is! If you go to https://app.keeptempo.com, you’ll see that you can’t just go right on in anymore, you have to specify an account, or go directly to the account URL you got in that email. For instance, we access Tempo at https://zetetic.keeptempo.com. If you are having trouble finding your account, please get in touch!
Before the upgrade, every user on Tempo had one unique profile, and worked with other unique users by grouping them together in various projects. The way Tempo works now is that users are grouped under a proper account, with an owner. For some Tempo customers, in particular for paying customers who shared projects with other paying users, they find themselves with more one account or domain that they can log into, and some of their time is in one account, and some of their time is in the other accounts.
To that end we have tried to make it as convenient as possible to use multiple accounts by hosting them on separate domains. That means you can keep a logged-in session to both your account and another person’s account in your browser at any time. You will still be able to run reports, log time, view data, etc.
Why did this change?
These changes were really critical for Tempo to continue operating effectively. As the system has grown there has been growing confusion on the part of new and existing users about the way that groups of people use Tempo. Previously there was no clear account owner, customized domain name, or centralized user administration, which added to the problem. Some companies even required multiple pay plans to support their needs, etc. While many of our customers had a clear understanding of the old “shared account” workflow, most people who tried out Tempo experienced problems setting up their company, creating projects and working with groups. Thus, this upgrade re-organized the model to make everything more consistent.
My time / co-worker / project is missing, help!
Don’t worry, your data is still here! We created an account for all paying users, and for all free users who had their own time outside of time logged to projects on paying accounts. Projects were moved under the account of the user who created them. If you would like us to consolidate your projects under one account, we’re ready and happy to help, please get in touch.
Am I being charged more now?
No! No one is being forced to upgrade, we’ve grand-fathered the plans of all our existing customers.
Do I have to upgrade because I have N users?
Nope! You may have noticed that our new plans charge users by how many people are in their account, as opposed to how many people they can have per project. Because this would be quite an unpleasant change for our existing customers, we decided to hook it up:
We calculated the total number of users you work with, added 2, and set that as the number of maximum users for your account. Previously, our plans were limited by people-per-project. So, if four was your limit, you could potentially have been working with 20 other people as long as they were on different projects (and your projects are unlimited). In that case you’d end up with a new limit of 22 users! For free!
Make sure you’ve updated the Site setting on the back of the widget to correspond with the new URL for your account, and that you use https! (e.g. we use https://zetetic.keeptempo.com). Make sure you leave off any trailing slash, or it won’t work (we’ll get to fixing that in an update to the widget).
Is the bookmarklet broken?
You just need to grab the updated bookmarklet. Log into your Tempo account, go to the Time screen, and grab it from the Time Entry form at the top of the screen.
How do I get to the mobile version?
For some mobile devices, we provide the mobile view automatically, but for some newer or unrecognized devices, you can click the ‘Switch to Mobile View’ link on the login screen. We will be adding a link to the mobile view from inside the application very soon for logged-in users.
Another option is to go to https://m.keeptempo.com. You will be shown a mobile version of the account lookup screen. Simply tap in your domain name or lookup your account, and you will be redirected to the mobile time entry screen for your account.
Why does it say that my billing info needs to be updated?
Because there’s a bug that we’re working on fixing right now! We have a list of affected users, but feel free to send us an email anyway if you want to be kept abreast of this issue once it is resolved.
Can I request my own subdomain? Or change it my self?
Absolutely! If you are the account owner, log in to your account, and click the Account tab from the left-side menu. Click the ‘change domain’ link on this screen and you’ll be taken to the form for changing your sub-domain.
We will keep updating this post today if we identify more issues.
I love seeing the change summary after a large and major branch is merged into the master branch. I think this is our best one yet:
282 files changed, 11893 insertions(+), 12129 deletions(-)
We successfully deployed the Tempo update “Teams” last night. There are a few lingering issues that we’re working hard to resolve. If you notice anything funny, please get in touch so we can get to work on it.
Tempo will be unavailable from the web this Saturday evening, November 21st, at 9pm EST / New York time, while we apply a significant update to the service. Time entries sent over Twitter or Email will be queued up and processed when the service comes back online. API access and the Mac OS X Dashboard widget will not be available while the system is unavailable.
We will post updates to the blog and the Tempo Twitter account about our progress Saturday night for those of you who wish to keep track.
If you have any questions or comments, please don’t hesitate to get in touch!