Updated, below: Apr 18 10:00 AM

On March 1st, we released Strip 1.5.0, largely a pretty successful release with a low number of bugs, crashes, and some big improvements that were long over-due. However, in that same release, we started using Apple’s In-App Purchasing system, and it’s caused a lot of confusion and frustration for our current customers. That’s really the last thing we wanted to do, as our customers are particularly great, so this post is about what happened, what we’ve done so far to fix it, and what comes next.

The Issue At Hand

Thousands of people have already paid the full price for Strip, $9.99 USD. We wanted to make sure that when they upgraded, they weren’t charged again by the new In-App Purchasing system in Strip 1.5.0. To ensure this, the program looked for a previous install on first launch, and made sure not to ask the customer to pay for an upgrade, which worked—on the upgraded device.

It didn’t prevent users from being charged for the new version on new or replacement devices where the old version of Strip wasn’t already available on the device for detection by the new version. Thus, anyone upgrading to a new device, doing a restore, replacing a device, or buying a new device was faced with being charged a second time, because they no longer had the old version on the device in question and could only download the new version. To make it worse, there appears to be no way for us to remedy this with the App Store and the In-App Purchase system as provided by Apple.

To add to the confusion, purchases made once in the iTunes store don’t have to be made again on the same account, they are free, but the iTunes store still presents it as a purchase until you agree to pony up the cash. Only then does it inform you that you get the new download for free, so you’re always trying your luck. But the In-App Purchase for Strip is considered a different “Product” in the iTunes App Store, and buying the old version didn’t make the new version free on a new device. Many users in this situation confirmed the purchase when prompted, expecting it to be free thanks to their previous purchase experiences, and were charged anyway. To these folks, it looked like we’d performed a bait-and-switch (or just screwed up).

What We’ve Done So Far

Answering everyone’s email and getting to the bottom of the various problems wasn’t going to be enough, so we got to coding.

We published Strip 1.5.1 to the iTunes App Store on Monday, April 4th. It fixes a couple of bugs in 1.5.0 (thus the delay in getting it out), so we recommend you install it now. This version has a facility allowing us to remotely grant a user unlimited access to Strip on their device. When a customer gets in touch and it turns out she has or will be double-charged, we can put their UDID in a remote database and give her a couple of steps to perform in Strip to cause a remote HTTP request to check with a little web service we set up for authorizing instances of Strip.

Back to Paid Downloads

We will be publishing Strip 1.5.2 as soon as it’s approved by Apple (we’ve already submitted it for review to the App Store). It contains a one-line adjustment to disable the in-app upgrade completely, and we’re going back to what works: $9.99 to download, no more screwing around. Anybody who upgrades won’t be charged for the download.

We’re really grateful to our customers for being so patient with us while we worked through all the email and got back to everybody. If you are currently in a pickle yourself, get in touch, and we’ll pull you out of the brine. We’ll post a note here and to the mailing list once Strip 1.5.2 is available for download.

Update: Strip 1.5.2 is now available in the iTunes App Store, all customers are advised to install this update.

Updated, below: Apr 8 9:22 AM

We first released our Password manager, Strip, to the iTunes App Store in May of 2009. It started life as a paid application priced at $9.99 USD to download. Unfortunately, this left early users with no way to test or evaluate the software before buying. Or course, the app was quite new to the store, didn’t have very many reviews, and users were reluctant to pay sight-unseen.

Like many application developers we decided to make a “Strip Lite” version that would provide an easy way to evaluate the software. This was basically the same application with a couple of build settings changed to introduce reasonable demo limits (10 entries, no WiFi Sync feature). Strip Lite allowed a potential customer to see if the software was indeed right for her before purchasing. At the time there was no other way to provide trial-ware, where the user has a certain period of time to review an app before paying for it.

Even so, there were some clear disadvantages to the Lite-app approach.

• All of our stats, ratings, and reviews were walled off into two separate gardens in iTunes Connect, one for each app.
• We had to develop a custom upgrade system so people could bring their data along to the paid version. Even though this worked, the process was a bit confusing and increased our support load.
• We would often hear from people who were confused about the versions, upgrades, differences and prices between the two application versions—another increase in support cost.

Strip Lite really did help grow the visibility and popularity of Strip by a wide margin, but it wasn’t ideal.

Demo-to-Paid using In-App Purchasing / StoreKit

When Apple announced In-App Purchasing for free apps it was widely touted as the perfect way to provide demo-to-paid upgrades. Indeed, it seemed like the ideal solution to our problems: we could make Strip free to download (with the Lite limits in place), allow users to purchase the full version inside the application, and get rid of Strip Lite and it’s confusing upgrade process.

Before taking the plunge on our most popular applicatiom, we tried it out on Codebook first, a paid-only app that had a smaller user base. It led to a big increase in overall downloads and sales, and amazingly we received no complaints. This seemed to be the best approach for us and our customers, so we got to work on building it into Strip for release in version 1.5.0.

On March 1st, 2011 we released the new version of Strip to the app store. At first there were no problems. On the contrarary, the jump from paid to free resulted in a huge spike in downloads, placement in the top-ranking productivity tools lists, and even an uptick in conversions. Unfortunately, this didn’t last very long, and the troubles started shortly after.

Is It Really Free?

Free apps with In App Purchases can be confusing on two levels.

On the one hand, even though we had placed a prominent note on the app description saying that Strip was only free to use for up to 10 entries without paying for the upgrade, many people don’t carefully read the descriptions. This resulted in some customers feeling that the In-App Purchase was a “bait-and-switch” tactic, even though this was never our intent.

On the other hand, I can’t tell you how many emails we received that asked, “is it really free to install on another device?” With a paid application, people are accustomed to selecting apps they already own in iTunes for installation on a new device with no charge, getting all the functionality immediately. However, when customers buy an In-App Purchase for your app on one device, it does not automatically transfer to a new device. Even though they are entitled to that purchase for free the user is asked to confirm if she would like to buy “1 Strip Unlimited” (love that qty of 1) for $9.99, with no hint of “free for you”. Only after the customer agrees to pony up the additional $9.99 does iTunes inform her that it will be a free download and that she won’t be charged¹.

Which version is this?

“Where’s the full version of Strip, I only see the Lite version available for the new upgrade!?”

When thousands of our existing customers saw that the only available upgrade for the new versions was free, they thought we’d eliminated Strip in favor of Strip Lite, along with their unlimited functions. This resulted in a big increase in support mail. This is a confusing thing to have to explain. We tried to head this off at the pass, by making this clear in our release announcements, but it didn’t help.

Don’t get us wrong, we love the opportunity to provide great support to our customers (if done right, these interactions are invaluable), but we really don’t like confusing them and wasting their time. We’d rather save that karma for when there are bugs—there will be bugs and you don’t want to use up good will!

But the next problem was a real Charlie-Foxtrot.

Double-Charged!

For users who’d already paid for the full version of Strip, we needed to make sure they weren’t charged again via In-App Purchase when they upgraded. Thus, when Strip 1.5.0 launched for the first time on your device, it would do some clever stuff to check for a previous install and grant you the unlimited upgrade by writing a receipt file onto the device in the application’s documents directory.

Unfortunately, that only works for one device. When a user gets a new device, the old version of Strip isn’t already on there, and only the new version where you have to pay in-app is available in the store. Since there is no old version on there already, it asks the user to pay again. This doesn’t just affect users who go and buy an iPad to go along with their iPhone, it affects people who need to get their devices replaced, or who upgrade and find iTunes just forgot to copy their apps and data over. Consider this lovely email from one irate customer who wiped his phone and couldn’t restore from backup:

"what the fuck iz wrong with you guys?? i bought the app 4months back. and now the app is askin me to pay another 10bucks for sum shitty upgrade. jus fix this issue asap."

This was an aberrational response, most of our customers are very good spellers (and rather polite), but it was a harbinger of serious trouble. At first, we had no way to help these folks get Strip without being charged a second time. We also had no way to tell the StoreKit API that the user should be granted the purchase for free when we did the grand-fathering check described above.

This problem was made worse by the confusion around the in app purchase screens. Now, when our customers downloaded the new version and then contacted us to ask, “Will I be charged again?” we had to go through a complicated process to determine what version the user previously had, when he or she made the purchase, what state the device was in (is this a restore?) and whether or not he or she would be charged before we could say, “go ahead, it’s gonna be alright.” Many customers just went ahead with the purchase, thinking they wouldn’t be charged per usual iTunes store policies (expecting the little “don’t worry it’s free” modal dialog to come up) only to find out otherwise.

The kicker here is that there’s no way for us to offer refunds in the iTunes Store, still, after all this time. Not cool at all, and not how we like to do business. The only thing we could do was to offer a $10 gift card to cover the re-buy, but since iTunes had already taken their cut, we covered 30% of the refund straight out of our own pocket. Even worse, we could only do that for our customers in the US.

A Temporary Hack

We published Strip 1.5.1 to the iTunes App Store on Monday, April 4th. It fixes a couple of bugs in 1.5.0 (thus the delay in getting it out), so we recommend everyone upgrade now. This version has a facility allowing us to remotely grant a user unlimited access to Strip on their device. If a customer gets in touch and it turns out she has or will be double-charged, we can put their UDID in a remote database and give her a couple of steps to perform in Strip to check with an authorization web service.

At least this allows us to help our customers out in the short term, but it still requires a lengthy support process full of confusion and often beginning with frustration for the user, so we’re not stopping here.

Back to Paid Downloads

We will be publishing Strip 1.5.2, which disables in-app purchasing, as soon as it’s approved by Apple. We’re going back to doing what works: $9.99 to download, no more screwing around. Anybody who upgrades won’t be charged for the upgrade a second time.

We won’t be using the In-App Purchasing API for this purpose again. In-app purchases clearly work well for icon packs, new levels in video games, and paid content downloads. They might even be appropriate for demo-to-paid apps that start out free, if you can live with forcing your users to “re-buy” every time the install on a new device (where they must confirm a new purchase before iTunes offers it for free). But for an app like ours there is just no easy and clean way to take care of the problems described here without incurring significant support overheard and/or creating a complex StoreKit companion to track purchases and authorizations.

We’re really grateful to our customers for being so patient with us while we worked through all the email and got back to everybody. If you are currently in a pickle yourself, get in touch, and we’ll pull you out of the brine. We’ll post a note here and to the mailing list once Strip 1.5.2 is available for download.

^[1] Suggestion for Apple: The StoreKit makes no effort to check before hand if the user is already entitled to the purchase, but it could do so quite easily, rather than requiring us developers to set up our own system. It could simply be part of displaying the price: just make it zero if the user already bought the requested product ID. Better yet, create an API method we can call to check. Don’t make us set up a companion system to register our customers, that’s the whole point of the iTunes App Store.

Update: As mentioned by Mark J below in the comments, the StoreKit API gives the developer a way to restore a user’s previous in-app purchases without forcing anyone to try their luck, with the SKPaymentQueue method - (void)restoreCompletedTransactions.

Site maps are one of the few things you can do to a website that don’t veertowards the “snake oil” side of SEO. Generally with these things, having directsupport from the Search Engine vendors is a straight arrow. In the process ofmaking our sites more accessible to customers, we noticed a void of thesewonderful little files.

The sitemap specification is so simpleit hurts. You have a list of URLs. They have a few properties. This isn’tparticularly surprising; Google, the inventor of the specification, is prettyskilled at applying KISS philosophy when it comes to web technology.

The number of sitemap tools onrubygems.org didn’tsurprise me either. After all, this is a very useful tool for people involvedin the web, and like it or not, a great deal of the energy thrown at ruby isfor web development.

That said, the ruby community loves its static site generators:theyreallylikethem.Needless to say, I was genuinely surprised when I came to the realization thatnot a single one of these sitemap systems supported them.

Enter Cartographer, which is astandalone system for generating sitemaps. In particular, it has three mainfunctions:

• Import an existing sitemap into an internal structure to represent it
• Find, with optional filtering, all the items in a given path and add them to to the sitemap.
• Generate the sitemap XML.

It brings along with it a few bonuses:

• Not attached to rails, rack, or anything other than Nokogiri, HAML, and Ruby.
• Easy to manipulate, no need to feel locked into a framework.
• Designed for static trees.

And a few disadvantages:

• Does not know about your routing, rewrite rules, or any other URL manipulation.
• Is not an “end to end”, pluggable tool. You need to invest some effort.

Cartographer makes heavy use of the ‘Find’ library that comes with Ruby, and assuch leaks its functionality via the add_tree call. For example,we use staticmatic and do the sitemap generation in an at_exit hook (so it’sthe last thing that runs, without getting into the nitty-gritty of rubyinternals.)

This Find leak is critical to the functionality of Cartographer and yields whatI believe to be very effective and succinct methods for dynamically adjustingwhat will be automatically included in your sitemap. For example, from theSTRIP Password Manager’s staticmatic generator:

You can see in the add_tree block we case over thepath with several regexes applied. Returning nil will callFind.prune which says, “Please do not look down this treefurther”. Cartographer will also refuse to include that tree into the sitemap.

So, here, we actually prune all non-html assets, and a couple of files whichunfortunately make it into our repos from time to time. We don’t want to indexour Search Engine instructions either. Also, for index.html files we prefer theparent path with a trailing slash, e.g., /demo/index.html werewrite to /demo/. If none of these criteria match, we simplyreturn what we got and it is added to the URL list verbatim.

The result looks something like this (trimmed for brevity):

Documentation for Cartographer ishere, and the code is ongithub. Please feel free to fork andadd suggestions, patches, or issues!

Strip 1.5.0 has been released and is available now from the iTunes Store as a free update. We think this is the best release of STRIP yet. It contains myriad small fixes to enhance the user experience and correct application behavior and a few more impressive updates:

• Support for staying unlocked on multi-tasking devices
• Support for landscape-orientation for easier data entry
• Updated graphics for Retina displays
• Updated SQLCipher to version 1.1.8

The various other updates:

• In-app upgrade to unlimited (no more STRIP Lite)
• Current customers who bought STRIP before 1.5 are grandfathered
• Field behavior changes fixed to take effect immediately
• Database info screen now includes Ditto replica info
• Password Generator sets controls to last-use settings
• Last-use settings are stored encrypted in SQLCipher
• Fixed display of lower distenders in fields values
• Field labels are no longer lower-cased on display
• Fixed default icon display for newly imported entries and categories

As you can see from the second listing above, we’ve made a change to how we allow people to try STRIP for free before making a purchase. Nothing has really changed for our existing customers, who are grandfathered to ensure the app is not limited in anyway.

Strip Sync update required

This version of Strip, 1.5.0, is incompatible with earlier releases of Strip Sync. As of today, an update is available for both Strip Sync for Mac OS X and Strip Sync for Windows. If you already have Strip Sync installed, simply fire up the program and it should offer to update and relaunch. If not, or if you’d like to install the software directly yourself, you can download the updated version below:

• Download Strip Sync for Windows v1.1.4
• Download Strip Sync for Mac OS X v0.2.5

As always, if you have any questions or run into any issues, please get in touch.

Databases and EBS: What you need to know.

Just a few things that you should know about EBS.

EBS is slow

All your data travels over a network before it reaches a disk, or data from thedisk reaches your instance. This means that writes and reads can be slow orintermittent at times.

Further compounding the issue, your SAN is shared with hundreds (thousands?) ofother users! While these machines are some high powered “big iron”, it stillmeans you’re going to have I/O contention and a number of other issues.

Even further, your disk access is metered! This means all those operations aretickling tiny little counters. This isn’t a lot in reality, but it all adds up!

On the bright side, EC2 instances have a lot of RAM. Let’s play to the field!

PostgreSQL Configuration and Use

This might sound a little preachy and redundant, but here goes:

Indexes!

No database should go without being properly indexed, from head to toe, witheverything you query upon and the vast majority of the combinations you use inyour queries. Yes, write performance will suffer, but we’re about to renderthat much less troublesome by sending the writes to RAM as frequently aspossible.

Less time spent searching tables = less disk access = greater performance.

Build queries to be sent over the network

If you’re doing anything with an ORM, you’re probably guilty of this at leastonce or twice: building your queries to be sent to the app to be handled later.You know those kooky DBA types that say “do everything in the database”, well,they’re on to something here.

Well, ‘lo and behold you do something like this:

When something like this:

Would have not only likely saved you a lot of computational cycles, but quite abit of network traffic is reduced, and continues to pay off as your tables growin size. This happens a lot in the rails community, unfortunately.

(Yes, I’m aware this example is a bit contrived. You could easily prepare thatquery with find() or ARel’s composition methods.)

The skinny: the less you do in the database the more you’re spending on networkresources and time to deliver your result. The database is probably working Ntimes as hard, too, to deliver your responses.

Even if it takes the “pretty” out of your code, do it in the database.

Shared Buffer Cache

Shared Buffer Cache is the meat and potatoes of PostgreSQL tuning. Increasingthis value will greatly decrease the frequency at which your data is flushed todisk. An EC2 Large Instance will happily accomodate a 4GB PostgreSQLinstallation which would be more than enough for lots of reasonably traffickedapplications.

Why is this important? The less time it spends writing to disk, or the lessfrequently it writes to disk, can mean a lot for your application’sperformance!

Database backups on the cloud

We have a few options for backing things up. As usual with redundancy, the bestoption is to… be redundant. (See what I did there?) Using a strategy thatallows us the best of both worlds.

EBS snapshots

You’ve already seen our snapshot script:

Which iterates over your volumes and maintains the last 5 backups.Here is a detailed account of the script’s function.

We use the script, amongst other things, to back up our database partitions,which are composed of the database master, the transaction log, and the backupsof the WAL.

WAL archiving

Write Ahead Logging and Continuous Archiving for Point in Time Recoveryis a pretty sticky topic and you would do yourself well to read that whole document.

Instead of repeating it here verbatim, I’ll tell you what our backup script does:

This script manages the archiving of three tarballs:

• base.tar.bz2, the base database system
• full-wal.tar.bz2, the whole WAL for the last day.
• pit-wal.tar.bz2, the point in time portion of the WAL.

The major difference between ‘full-wal’ and ‘pit-wal’ is that at the time thefirst backup is taken (the night of the backup), the data may not be fullycommitted to disk. Therefore, we write as much as we can to the ‘pit-wal’ filefor the purposes of crashes that day. The ‘full-wal’, as you might suspect, isthe fully written representation and is actually written out a day after thebackup occurred.

In a recovery scenario, both of these tarballs would be merged with theexisting WAL files in order of ‘pit-wal’, then ‘full-wal’ would be unpacked.

The WAL directory itself has some data hidden in the filenames, let’s checkthat out:

2011-02-01 09:05 000000030000000300000026
2011-02-01 09:05 000000030000000300000026.000076B8.backup
2011-02-01 10:12 000000030000000300000027
2011-02-01 11:30 000000030000000300000028
2011-02-01 12:57 000000030000000300000029
2011-02-01 14:10 00000003000000030000002A
2011-02-01 14:58 00000003000000030000002B
2011-02-01 15:30 00000003000000030000002C

The filenames themselves hold two important pieces of information:

• The first 8 characters of the filename are the recovery version. As we’re good little children and test our backups, this is at version 3.
• The last 8 characters of the filename are ordered, you can see this by comparing the times and the filenames themselves.
• If there is an extension, that is a demarcation point where pg_start_backup()/pg_stop_backup() was invoked. This is what we use to create the ‘full-wal’ tarball.

As for the backup structure? Well, here’s a sneak peek:

2011-01-28 09:05 2011-01-27.09:00:01/
2011-01-29 09:05 2011-01-28.09:00:01/
2011-01-30 09:05 2011-01-29.09:00:01/
2011-01-31 09:05 2011-01-30.09:00:01/
2011-02-01 09:05 2011-01-31.09:00:01/
2011-02-01 09:07 2011-02-01.09:00:01/

The $today and $yesterday calls just generate these filenames. At the endof the script, we see this idiom:

cd $backup_dir
ls -1d * | sort -rn | tail -n +15 | xargs rm -vr 
cd $OLDPWD

Which is a way of saying, “show us the last 15 dirs and delete the rest”. Thiskeeps our filesystem size low and we rsync these files nightly.

The sed usage here is a little tricky but not anything incomprehensible. Basically,

breakpoint=`ls *.backup | sort -r | head -n1 | sed -e 's/\..*$//'

Finds the latest backup file. Now,

arline=`ls | sort | sed -ne "/^$breakpoint$/ =" `
archive=`ls | sort | head -n $arline`

Uses that as a demarcation point to determine the archive files. Those filesare archived and removed and result in full-wal. The rest leftover result inpit-wal.

Happy Hacking!