This has been cross-posted from our discussion forum.

There are two options that applications commonly use to manage key material when using SQLCipher:

1. Password or Passphrase - An example of ths type of integration is an application that requires the user to enter password or PIN number with a device keyboard, and the input is provided to SQLCipher for use as a key. SQLCipher includes special functionality to derive an encryption key from user provided input.
2. Raw or “Random” Key Material - In some advanced situations an application might generate random key material for a database (i.e. random bytes) which is separately secured by an enclave, key store, hardware security module, or otherwise encrypted. The application provides this random data to SQLCipher for use as a key.

In the interest of interoperability it is strongly recommended that any key material passed to SQLCipher consist solely of valid UTF-8 encoded strings that do not include zero byte values internally. This recommendation applies universally to all keys, whether random or not, and no matter how they are supplied (e.g. sqlite3_key, PRAGMA key, or any wrapping API).

With Option 1, it is a given that the key material will be a string value because it is entered by the user. However, Option 2 requires additional consideraton on the part of the developer. While it may be technically possible to generate random bytes in code and coerce them to behave like strings in some languages, it must be avoided due to the potential for unpredictable behavior and compatibility problems, for instance:

1. diffuculty opening a database using certain tools with such a key (e.g. one can’t provide raw binary values via SQL statements (e.g. PRAGMA or ATTACH), the SQLCipher command line shell, database GUI application, or using certain programming languages)
2. corruption of key material due to encoding or decoding operations that encounter invalid code pages
3. truncation of key material passed through languages or APIs rely on a NULL terminators when zero-bytes are present in the random key material

For these reasons, SQLCipher has standardized on Raw Key syntax since it’s initial release. It is the supported method for using binary data directly as key material. A Raw Key is simply a fixed length byte array with a standardized BLOB literal representation and hexadecimal encoding that is interpreted by SQLCipher and used directly as an encryption key. Using this feature an application can safely generate random key material and instruct SQLCipher to use the exact 32-byte (256 bits) sequence as a key, for examle:

PRAGMA key = "x'2DD29CA851E7B56E4697B0E1F08507293D761A05CE4D1B628663F411A8086D99'";

Using this approach it is the calling application’s responsibility to ensure that the data provided is a 64-character hex string in BLOB format (i.e. x'hexvalue'). As an added benefit, SQLCipher will bypass the KDF when a Raw Key is provided which can result in a substantial performance enhancement when opening databases.

If it is not possible for an application to use the Raw Key syntax for some reason, it is still the application’s responsibility to guarantee that the passphrase or key material consists of a valid UTF-8 string. This can be accomplished either by converting a random byte to a UTF-8-safe string using an intermediate encoding like Base64, or by constructing the random key from a set of valid UTF-8 code pages.

Finally, for applications that may already be using potentially invalid keys for some subset of users, we recommend converting key material going forward using the “rekey” feature to re-encrypt the database with a new key that is of a valid format, i.e. either a Raw key or a valid UTF-8 string.

Our password manager Codebook got a nice shout-out on the Mac Geek Gab podcast from MacObserver.com on a recent episode, our thanks to them and whomever brought us to their attention! It’s an enjoyable weekly show with a lot of great tips, and being Mac buffs ourselves it’s a nice forum for us to appear in. We work hard to make sure that Codebook is a good Mac app, conforming to the interface standards while providing features that power users expect on the platform (our eyes are peeled to this week’s exciting announcements from WWDC). You can catch the bit about Codebook 29 minutes in by listening here, or subscribing on iTunes.

SQLCipher 4.2.0, which includes the following important changes, is now available:

SQLCipher Core

• Adds a new PRAGMA cipher_integrity_check which performs an independent verification of page HMACs
• Updates SQLite baseline to upstream SQLite 3.28.0
• Improves PRAGMA cipher_migrate to handle keys containing non-terminating zero bytes

Availability

Commercial Edition - On-demand access to new releases of SQLCipher Commercial Edition are available to licensees with an active CipherCare subscriber subscription, along with private, prioritized support directly from Zetetic. CipherCare subscribers will receive a separate email notification regarding the update and can contact us to request the latest SQLCipher distribution and applicable software license codes.

SQLCipher Enterprise Program - Enterprise Program Subscription customers will receive a separate email notification about the release, and the latest SQLCipher packages and license codes will be provided directly via your organization’s private online software delivery share.

Community Edition - SQLCipher 4.2.0 in source format is directly available on GitHub. The Community Edition of SQLCipher for Android (4.2.0) is available via AAR packaging. The Community Edition of SQLCipher for iOS can be compiled from source or using CocoaPods.

SQLCipher 4.1.0 is now available, the first minor release in the 4 series of the library. This update to SQLCipher 4.1.0 includes the following important changes to the library:

SQLCipher Core

• Based on upstream SQLite 3.27.2 (current latest)
• Defer reading salt from database header until key derivation is required
• Add PRAGMA commands: cipher_settings and cipher_default_settings to query current runtime configurations
• Disable backup API for encrypted databases
• Deprecate the following PRAGMA commands: fast_kdf_iter, cipher_hmac_pgno, cipher_hmac_salt_mask
• Improve sqlcipher_export routine and restore all database flags
• Clear buffer if cipher operation fails

SQLCipher Core Details

SQLCipher now defers reading the salt from the database header until key derivation occurs. Previously, SQLCipher would read the database header when the codec was attached to determine the per database salt. Unfortunately, this could present a problem where a client library which creates multiple connections to the same database file that do not initially exist. When the first connection triggers a codec to attach, SQLCipher would attempt to read the database salt from the file, or create a salt when not present, however the salt is not written to disk at this point. Subsequently, if a secondary connection is used with the same database file, when the codec attaches and the salt value determined, a different salt value will be created if the first connection hasn’t written to disk. SQLCipher now defers determining a salt value from the database until the key derivation process occurs, this prevents the scenario above from occurring.

Two new convenience PRAGMA’s were added that target the runtime configuration settings for SQLCipher. The new PRAGMA commands cipher_settings and cipher_default_settings allow a user to read at runtime the exact configuration settings used for the current connection.

The SQLite backup API has never been supported within SQLCipher for encrypted databases due to incompatibility of the API within the context of SQLCipher. We now disable this feature when the database is encrypted. Alternatively, the sqlcipher_export(...) convenience function may be a solution for those looking for a backup of an encrypted database.

The sqlcipher_export routine has improved sourcing of table naming which addresses scenarios where the sqlite_master table are modified externally by users. The database flags are fully restored now following the completion of the export process.

Finally, we have deprecated several PRAGMA commands which will be removed in a future release of SQLCipher: fast_kdf_iter, cipher_hmac_pgno, cipher_hmac_salt_mask.

SQLCipher for Android (4.1.3)

• Add support for keying database from byte array
• Fix to release the lock when an exception is thrown

SQLCipher for .NET, Xamarin, and Windows

For client developers integrating with the popular sqlite-net client library, a fix is in place for those using the asynchronous variant of the library via SQLiteAsyncConnection which addresses a situation where creating a new database file may have caused an error in some situations.

Availability

Commercial Edition - On-demand access to new releases of SQLCipher Commercial Edition are available to licensees with an active CipherCare subscriber subscription, along with private, prioritized support directly from Zetetic. CipherCare subscribers will receive a separate email notification regarding the update and can contact us to request the latest SQLCipher distribution and applicable software license codes.

SQLCipher Enterprise Program - Enterprise Program Subscription customers will receive a separate email notification about the release, and the latest SQLCipher packages and license codes will be provided directly via your organization’s private online software delivery share.

Community Edition - SQLCipher 4.1.0 in source format is directly available on GitHub. The Community Edition of SQLCipher for Android (4.1.3) is available via AAR packaging. The Community Edition of SQLCipher for iOS can be compiled from source or using CocoaPods.

We’ve had an excellent and productive year working on Codebook in 2018. As we charge hard into 2019 working on our next major version update, Codebook 4, we’d like to take a look back at some of the improvements we’ve made to our favorite password manager over the last year.

Before we do, though, it should be noted: Codebook has been in the iTunes App Store for over 10 years, since 2008! Originally a reboot of the venerable Palm password manager that had some pull among an earlier generation of sysadmins, it eventually spread from iOS to Android, macOS, and Windows. A lot’s happened since then! I won’t get into it all, but if you’ve been with us all this time, a big hearty cheers from all of us at Zetetic! 🍻 Here’s what we’ve gotten up to in Codebook’s tenth year.

We released an absolute ton of regular updates over the last year for maintenance, operating system and device upgrades, and bug fixes across all platforms, too many to mention here. However, each version of the app now contains a Release Notes feature, providing up to date information about what’s new.

Fingerprint and Facial Recognition Improvements

Codebook for iOS has supported Touch ID authentication for some time now (allowing you to login with your fingerprint instead of your master password), but this year we added support for Face ID authentication on new iPhones and Touch ID authentication on newer MacBook Pro laptops. Also, by popular demand, we’ve added support for Fingerprint authentication on Android.

There is, however, one drawback to this convenience. A new user has only ever entered their master password twice (to set and then confirm it) before they are prompted to enable Touch ID or Face ID login during application setup. They are never required to enter it again until an unforeseen event causes it to be required (for instance, Touch ID can become unavailable for security reasons, like a new finger being added, or when restoring from an iCloud backup on a new device). By that time they may have forgotten it!

We’ve begun taking some steps to help mitigate this problem on iOS, including some more helpful warnings about it on setup, and a reminder that encourages the user to look up the master password and be sure they remember it or have it backed up somewhere safe. We’ve got more planned to help with recovery coming in Codebook 4.

Free Trials on iOS

This is a really big deal. Previously we charged up-front to download Codebook for iOS, meaning you couldn’t try it before you bought it. Well, you could, via the Lite version, but this was less than ideal in that it meant we had to necessarily limit some core features.

With the new ability to offer free trials and a pro upgrade in the App Store via In-App Purchases, and the ability to grandfather in customers who paid up front to download Codebook, we are able to offer a much better experience to anybody who just wants to download the iOS app and try it out before they invest. AND, we were able to offer a smooth upgrade process without confusing or alarming our existing customers about the change in licensing. We appreciate all the help our beta testers gave us with this, and the patience our customers showed when there were hiccups.

This also let us retire the Lite version in the App Store, along with the iPad-only version that had been discontinued. Having just one app in the App Store for all iOS users is less confusing, and less work for us.

Improved Trials on macOS

Codebook for macOS got a solid update to trial behavior as well, in the direct version (the Mac App Store version has no trial mode). We distribute the direct download version as a 14-day free trial, like we do on Windows, but we made some really nice changes:

• Brand new and improved UI for the trial window
• The trial now tracks the number of days that the app was actually used, making the trial a bit more forgiving for someone who hasn’t really had a chance to try the app out
• When the trial expires the app is still functional, in read-only mode. Editing and sync features are disabled, but all other features and the user’s data are still available, passwords can still be filled in with Secret Agent, etc.

These are only some of the changes we’ve been making in order to improve the initial user experience with Codebook. This work benefits our existing customers, too (for instance, you can restore your data onto a new device during setup!)

Improved Downloads for macOS and Windows

People have reported frustration hunting around the website and discussion forum looking for direct download links to the macOS and Windows installers. It’s well-placed criticism, we didn’t want to make such links very public, lest someone download the apps without agreeing to our export compliance requirements! Some folks were signing up in the free trial forms to get access to the link and receiving follow-up emails about a trial they didn’t need. We’ve recently updated the macOS and Windows download pages to make it easier to download the app directly without having to sign up for a trial.

AutoFill Passwords on iOS

This is actually two interesting feature milestones! First we created an application extension called Find in Codebook that allows you to fill in passwords in Safari from the Share Sheet using the passwords you have stored in Codebook (with proper authentication, of course). This works well, including support for filling in TOTP fields, but would have benefited from better system-level integration with iOS.

And then iOS 12 introduced an API for password managers like Codebook to AutoFill Passwords, identifying login forms on websites and third-party apps. So, we did that, too! We were easily able to adapt and improve the UI of Find in Codebook to support AutoFill Passwords. This means that our iOS users can opt to fill in their passwords from Codebook right from login forms on web pages and third party apps. Fantastic!

Password Review / HaveIBeenPwned.com on iOS and macOS

One of the most interesting projects in password research and security is Troy Hunt’s HaveIBeenPwned.com service, and it really is that, a public service. It lets users check their accounts for inclusion in password and account breaches, and at this time contains the details of nearly 7 billion accounts (as I understand it, another large data breach is being added as I edit this!)

The website also offers an API that allows apps and other websites and online services several tools for looking up information about breaches and whether a particular password is included in a breach. In Codebook for iOS and macOS we added a feature called Password Review that uses the HaveIBeenPwned.com API to check if one of your passwords stored in Codebook has been seen in any breaches, and how many. If you have a password you think is super strong, and unique, you might still want to check anyway! Codebook does this without ever sending your actual password to the service due to a rather clever security model used by the API, but the feature is disabled by default and has to be enabled by the user.

Search Scopes on macOS and Windows

When you start typing in the Search field at the top right hand side of Codebook’s main window, the Entries view in the middle of the window is automatically populated with matching entries. Normally, the scope of this search is any entries that have a name matching the search term, and any entries that have a field that matches the search term.

There were circumstances were some customers wanted to be able to change that, and others who would have found it useful for entries that match a particular label name. And so, we implemented search scopes! It’s pretty handy when you want to customize your labels but are unsure exactly what entries are using what labels.

Added support for Dynamic Text and Large Type on iOS

Codebook for iOS got a big improvement in terms of accessibility when we adopted Dynamic Text and Large Type, adapting the UI to the user’s custom Accessibility settings in iOS.

Added new Magnify feature on iOS, macOS, and Windows

Sometimes we have data stored in Codebook like a phone number or a PIN that we want to display very large and prominently across the screen of the device, perhaps in order to enter it on another device or to view it across a short distance. The Magnify feature does just that, displaying the selected field in a modal, heads-up display, with a large, mono-spaced font. This feature is available in all four apps.

Wrapping Up

That’s the major hits, but as noted before, each app has its own Release Notes feature containing many more updates, adjustments, and improvements that may interest you. In addition, we also post to the discussion forum detailing updates to the individual apps as they are released.

It’s been a busy time, but there’s lots more in the works for Codebook 4, including the new sync system we have been working on and are near completing. We’re looking for new beta testers that would be willing to help review and test Codebook 4 and the new replication system, as well some other excellent new features like the recovery/rescue feature we’re working on. If that sounds like you, please sign up here! We have some serious alpha, dog-fooding testing to do first, but we’re hoping to start beta testing by May, and possibly as early as April.