Fixing a "Error copying keychain data" Problem with iTunes Backups

lunedì, 2013-02-04; 15:25:09



Today, dear reader, I have a story of intrigue, of frustration, and of redemption. Well, it’s really just about hacking up my iTunes local backups to fix a backup/restore problem I was having with my iPhone. If you’re looking for steps to fix the problem, skip to the bottom. If you’re interested in hearing what I tried that didn’t work and why I was stuck in this situation, though, keep reading.

After updating to iOS 6.1 on my iPhone 5, I started having problems backing up my iPhone to my iMac. I didn’t notice at first, because lately I’ve been using my iMac as a second monitor for my MacBook Air rather than an actual iMac.

By “problems backing up my iPhone”, I pretty much mean exactly that. My iPhone worked normally, but every time I would plug it into my iMac and attempt a backup, I would get a generic dialog helpfully explaining with verbiage similar to: “iTunes could not back up the iPhone because an error occurred”. This would happen regardless of whether the backup was via a full sync or merely by pressing the “Back Up Now” button in iTunes.

Further details on the specific problem I was having were found in the Console:

AppleMobileBackup[5219]: ERROR: Backup message response: 1 Error copying keychain data (MBErrorDomain/1)
AppleMobileBackup[5219]: WARNING: No MobileSync error code defined for error code: 1
AppleMobileBackup[5219]: ERROR: Backup error: -1

Alright, I guess that’s somewhat helpful in pointing out the problem, but not the solution. Since it was related to the keychain, I figured I’d clear out all my network passwords on my iPhone (via Settings.app, go to [General —> Restore —> Restore Network Settings]). I also tried removing both my password lock and my restrictions password. None of this had any effect: my iPhone would still fail to backup with the same keychain error. I also tried backing up to a different Mac, and that didn’t work either.

A post on Apple’s discussions site claims iCloud backups proceed successfully in this situation, even though local backups don’t. I tried doing this, but my iPhone estimated that the backup would take 9 hours, so I cancelled it. I think iCloud backups include keychain items, so I’m not sure why local backups would fail but iCloud backups would work.

In any case, I was annoyed, so I proceeded with a complete restore of my iPhone. I’ve got a fix that doesn’t involve restoring your device, but if you are planning to restore your device at this point, let me give you an important recommendation: back your device up to iCloud first, if it works. Please, please do this.

I found out the hard way that when your backups aren’t working due to this error, restoring your backups won’t work either! Arrgh. If iCloud backups work, then you can probably successfully restore your device from them. Unfortunately, I’ve never backed up to iCloud even once, so I couldn’t restore from there. And I couldn’t restore from my local backups either. I would get similar generic error dialogs, such as this one:

And in the Console, I’d get similar log lines:

AppleMobileBackup[5494]: ERROR: Restore message response: 1 Error restoring keychain: 1 (MBErrorDomain/1)
AppleMobileBackup[5494]: WARNING: No MobileSync error code defined for error code: 1
AppleMobileBackup[5494]: ERROR: Restore error: -1

Worse, this seems to be a latent error that was triggered by my update to iOS 6.1. Using Time Machine, I pulled iPhone backups from early January, and they had the same restore problem. Arrrrgh. Without an iCloud backup, and without any working local backups, I was stuck with an iPhone that didn’t have any of my data. I also have hundreds of apps on my iPhone (I know, I know), so reorganizing them would’ve been a real pain. I could restore from my iPad, which is set up similarly, but has different data and doesn’t restore home folder organization. (By the way, I updated my iPad to iOS 6.1 as well, and it hasn’t had these problems.)

The internets weren’t helpful in solving this problem, but I did find this page which helpfully explains the structure of iTunes backups. In particular, there is a single file which contains all the keychain resources: in ~/Library/Application Support/MobileSync/Backup/, you should see folders named with the UDID of your devices. Inside each backup, there is a file named 51a4616e576dd33cd2abadfea874eb8ff246bf0e. This is the file corresponding to your device’s keychain backup.

With that information, I tried deleting just that one file, and then restoring the backup. iTunes changed its tune and claimed the backup was damaged, and refused to restore it. I tried replacing it with a different file, or replacing it with a zero-length file, and the same thing happened.

That afore-linked page also explains the format of the Manifest.mbdb file that contains all the information for which file corresponds to what data, so it was time to go futzing around with that. Here, ultimately, are the steps that fixed my local backup.

  1. Connect your device to your Mac. Open iTunes, go to the Device Summary page, click where it says “Serial Number”, and iTunes should show you your device’s UDID number. Note this down.

  2. In the Finder go to ~/Library/Application Support/MobileSync/Backup/.

  3. Find the folder named with the UDID from step one, and duplicate it. Move the duplicate to a safe location: you want to have a pristine copy that you can always revert back to (even though it doesn’t actually work to restore). Better is having Time Machine enabled, so you have progressive copies of it, but you can’t enable that after-the-fact. Even if you have Time Machine backups, though, duplicate this folder just in case!

  4. Now you need a local backup of a different device. If you have a second iOS device, connect it to your Mac and perform a backup. (Make sure that if you encrypt your backups, you’ve used the same password for this backup as you do for your problematic backup.) Note down the UDID of this device using the method in step one, too. If you have an older backup that you can successfully restore with, you can use that as well, if you want to fix your newer backup. (You can also connect a friend’s device to your computer and back it up, or have them send you just the two files you’ll need from their backup — more on that later — but this method changes the keychain data in your backup, so you’ll end up with your friend’s passwords on your device. They probably don’t want that.)

  5. If you had a second device, skip this step. If you don’t have a second device, you’ll need to completely restore your device and perform a new backup. Here’s where you try an iCloud backup, if you haven’t before! Before you erase and restore, though, rename the folder ~/Library/Application Support/MobileSync/Backup/your-problematic-device-UDID/ to “your-problematic-device-UDID.bad”. Once you have a good backup, the new backup will have the same UDID — rename the folder to “UDID.good”, and rename the problematic backup back to the original UDID.

  6. Now, you should have two backups: one is your problematic backup that you are trying to fix, and the second is the good backup with valid keychain data. You created a backup of your problematic backup, right? OK, here we go. From your good backup, copy the file named “51a4616e576dd33cd2abadfea874eb8ff246bf0e” into the folder of your problematic backup. To be clear, you copy the file located here:

    good-device-backup-UDID/51a4616e576dd33cd2abadfea874eb8ff246bf0e

    to the same location in your problematic backup:

    your-problematic-device-UDID/51a4616e576dd33cd2abadfea874eb8ff246bf0e

    When the Finder alerts you that there’s already a file in that location with that name, replace it.

  7. OK, you’re going to need to do some hex editing. Go download Hex Fiend, a very competent, fast, free, and open source hex editor.

  8. In Hex Fiend, open the “Manifest.mbdb” from both your good and your problematic backup. Make sure you know which is which!

  9. In the Manifest.mbdb file of your good backup, search for the word “keychain”.

  10. The webpage describing the iTunes backup format lists the possible fields for every record that this file contains. You should see the string “KeychainDomain”. Highlight it, and you’ll notice that Hex Fiend’s status bar shows you’ve selected 14 bytes of data.

  11. If you highlight the character immediately before the string “KeychainData”, and set the data inspector to “Signed Int” or “Unsigned int”, it’ll say “14”. (If you don’t see the data inspector at the bottom of the window, you may need to enable it from the Views menu.) Note that this is the length of the string “KeychainData”.

    You should also see a string that says “keychain-backup.plist”. I tried changing that to make iTunes backup/restore a non-existent file, but it just gave me a different error.

    So, what we’re going to do is change the SHA 1 hash field of the keychain data in your problematic backup with the SHA 1 hash of the keychain data in your good backup. Since we already changed the actual keychain data itself in step 5, this should satisfy iTunes in verifying that the files contents are known and good.

  12. Somewhere after the string “keychain-backup.plist” there should be a blank character that has a hex value of 14. Highlight it, and the data inspector should give you a value of 20. Not coincidentally, SHA 1 hashes are 160 bits in length, or 20 bytes!

  13. Starting with the character immediately following the blank character, select 20 bytes worth of data. Copy it.

  14. Now, repeat steps 8 through 12 with the Manifest.mbdb file from your bad backup, but instead of copying the SHA 1 hash, you’ll paste the SHA 1 hash that you just copied from your good Manifest.mbdb file.

  15. Save. Now restore your device from this backup. It should work!

I suspect that a similar technique will allow you to replace other arbitrary data in your backups.


Tips   Permalink   Post a Comment

Generated Text Buttons

mercoledì, 2012-10-31; 00:08:40



Let’s say you have some UI elements in your iOS app. These elements have text on them! (Don’t they all!) But these aren’t ordinary elements! These are buttons with a custom styling, with the text designed to integrate with the style. Maybe you want recessed text, or floating text on a background, or some other effect.

Well, normally you’d have to get your resident designer to create all these assets, right? And then whenever the text changes, or you decide you want to tweak the effect, you have to get your designer to tweak their Photoshop file and export you another asset. To think nothing of localization, all the mounds of extra works that creates for your designer, and keeping track of all those files.

Well, if you hate the prospect of even thinking about all that, you’re in luck! Have I got the code for you! Want text masked to a background image, with anything behind it shining through? BAM!

Masked Text

Want text clipped from an image to put on top of some other background? BAM!

Clipped Text

Want recessed, transparent text? BAM!

Recessed, Transparent Text

Want glowing text? BAM!

Glowing Text

Want floating, wooden text? BAM!

Floating, Wooden Text

Don’t like the wood texture? BAM!

Floating, White Text

Want some custom crazy styling that even your MOM wouldn’t want to see in any self-respecting iOS app? BAM!

Crazy, Ugly Text

That’s right, boys and girls, today is your lucky day! All of the above images were generated purely in code, from fonts all included on your favorite iOS device. You can tweak the font family, size, shadow, gradient fill, and a bunch of other attributes. Best of all, this code also supports non-ASCII characters, Unicode characters, and non-Roman language characters (such as Arabic and Japanese), as you can see in the above examples.

If you’ve been looking for this your whole life, then just head on over to GitHub to download the code. BSD-licensed, of course.

(And remember, don’t abuse this code. It can be used both for good and evil!)

Acknowledgements

This article is the unofficial part two of a three-part series of code that my previous employer, Shiny Things, has graciously allowed me to open source. This code was super-useful in creating one of the early alphas of Sakura Quick Math, and for all I know, this code may still be in use in the current incarnation of the app.


Software Development   Permalink   Post a Comment

Offline Game Center Reporting

sabato, 2012-09-29; 14:35:09



If you’re developing a game in iOS and implementing Game Center support, you may have come across this paragraph in Apple’s Game Center Programming Guide:

Whether reporting achievement progress on a single achievement or multiple achievements at once, your game rarely needs to do anything specific when an error occurs. If an error occurs, such as when a network is not available, Game Kit automatically resends the data at an appropriate time.

This is a lie.

One of my pet peeves about Game Center when it was first introduced was that achievements and scores earned offline were never properly transmitted to Game Center. When iOS 4 was the current major version, the above excerpt in the documentation instead said something like, “if an error occurs, you must save the achievement yourself for transmission at a later date”.

Once iOS 5 came out, the current language in the guide was added in. Hooray! Game Center scores and achievements would be reported across all Game Center applications even when offline!

Unfortunately, this isn’t the case in practice.

I have an iPod touch. Frequently, I’m at home or some place that has Wi-Fi, but sometimes I’m not. Often, this happens when I’m on the train going somewhere, and I have 30 minutes or so to kill. So I open up my favorite Game Center-enabled game, and play for a bit. But it’s so annoying when you get a really good score or an incredibly hard achievement, because, in practice, automatic offline Game Center reporting never works.

I recently verified this problem on both iOS 5 and iOS 6. My iPod touch is a 3rd-generation device and iOS 6 doesn’t support it. I tested this supposed Game Center functionality by turning off Wi-Fi, getting a high score in Super Hexagon, and then turning Wi-Fi back on. Nothing I could do would cause the new high score to be correctly reported to Game Center. (I messed around with both the Super Hexagon app and the Game Center app itself.)

There’s a hedge in the documentation, which says that the data is resent “at an appropriate time”, but in all my use of iOS 5, I’ve never seen an offline score or achievement reported correctly. I tested iOS 6 on my iPad 3, with similar results.

It just doesn’t work. (I’ve filed a bug.)

So, how do Game Center developers fix this issue? Well, they can do what the iOS 4 Game Center Programming guide originally said: check for an error during transmission of the score or achievement, and manually resend it at an appropriate time yourself, as Game Kit said it would do.

Luckily for you, I have some spiffy code that does this all for you. Check it: https://github.com/simX/offline-game-center. Information on using this code is included in the repo. Any bug fixes, code suggestions, etc., are heartily welcomed as pull requests on GitHub, or via Twitter or e-mail.

Sakura Quick Math and Shiny Things

Please note: I wrote this code for my previous employer, Shiny Things. At the time I left the company, the iOS game was unreleased, but I was graciously allowed to open source just this part of the code so that other developers could implement offline Game Center reporting easily. Initially, I decided not to post the code, because iOS 5 had this functionality built-in, but after further use of iOS 5 and 6 and determining that this functionality was not actually working, I finally polished up the code and wrote this post.

This past week, Sakura Quick Math was released. The UI has been completely revamped since last I saw it, but, thankfully, my code to submit offline Game Center achievements is still present! So, if you’d like to test this functionality in a shipping app, go get Sakura Quick Math.

As far as I know, Sakura Quick Math is the only iOS game that correctly reports Game Center achievements and scores when they are obtained while offline.

So, many thanks to my former employer for allowing me to open source this code!


Software Development   Permalink   Post a Comment

Entitlement

sabato, 2012-03-03; 13:31:18



A few days back, The Oatmeal published this comic about how easy and convenient it is to watch TV shows by pirating them, compared to the hassle of watching them legally.

Andy Ihnatko filed this reaction:

The single least-attractive attribute of many of the people who download content illegally is their smug sense of entitlement.

He concludes:

The world does not OWE you Season 1 of “Game Of Thrones” in the form you want it at the moment you want it at the price you want to pay for it. If it’s not available under 100% your terms, you have the free-and-clear option of not having it.

Ugh.

Marco Arment has a decent response. He says:

Relying solely on yelling about what’s right isn’t a pragmatic approach for the media industry to take. And it’s not working. It’s unrealistic and naïve to expect everyone to do the “right” thing when the alternative is so much easier, faster, cheaper, and better for so many of them.

The pragmatic approach is to address the demand.

This addresses Ihnatko’s conclusion, that you have the “choice” of not watching video that you can’t acquire on desirable terms. But calling out the “smug sense of entitlement” of consumers is annoying and arrogant. Here’s why.

Hulu, iTunes, or any other video streaming service would not exist had it not been for the advent of piracy.

Media companies have been dragged kicking and screaming into the digital age. They were dragged kicking and screaming into the analog age. When VHS came around, they tried to sue it out of existence. They tried to subvert the unencumbered CD standard to add copy-protection, even though it broke compatibility with many existing devices. When MP3s started gaining in popularity, they whined about music sharing services, and again lined up their lawyers. When digital downloads started coming of age, they encumbered them with onerous DRM. When iTunes became the #1 music store in the world, they whined and complained that people weren’t buying albums anymore, even though most of the songs on most albums were shitty filler songs that people didn’t want. They got Apple to raise prices on digital downloads, even though it’s far cheaper to distribute digital downloads than physical CDs, and the media companies get most of the profit anyway. They create ridiculously onerous licensing terms for artists, on a country-by-country basis, making it difficult to legally acquire music from other countries. DVDs are often stupidly region-restricted, meaning that the foreign film you want to get from Italy? Nope, that’s right, it won’t play in your DVD player! They blackout local sports games because some company has exclusive rights. They prevent you from watching international video for no good reason at all. They make you sit through stupid FBI warnings and un-skippable previews.

With the exception of the creation of the compact disc (which they later regretted), media companies fight tooth and nail against consumers at every step of the way.

Had it not been for computers that make it easy to transcode digital video, and pirates who are willing to flout the law to get TV shows whenever they want, none of the current legal options would ever exist. Media companies would have been happy to infect all CDs with copy-protection. They would have been happy to force you to continue buying albums instead of singles. They would be ecstatic to continue making you pay again and again for the video you want even though it only takes a small amount of computing power to get the video in the format you want it in. They would play 10 advertisements for every minute of video if they could. They would wish streaming video out of existence.

You know who’s fucking entitled?

Media companies.


Rants   Permalink   Post a Comment

PackageMaker and Installer

mercoledì, 2012-02-15; 03:01:07



Drag-and-drop installation is a great method of software distribution. For software that is self-contained and doesn’t require living in any specific directory to run, it allows the user to be as neat or as messy with their hard drive hierarchy as they see fit. But for certain types of software, certain things need to be in certain places, and drag-and-drop installation just doesn’t cut it. ClickToFlash, a Safari plug-in I worked on a few years ago, is one of those cases.

Really, the only two types of software distribution for Mac OS X that are at all credible in my mind are 1) drag-and-drop installation, and 2) Apple’s installer technology. Any other method of installation is dead on arrival. There are numerous criteria — remote deployment, user familiarity, flexibility, scriptability — areas in which these two methods excel across the board. All other installer technology (VISE, FileStorm, I’m looking at you) falls flat in at least one of these areas, if not more.

Despite that, however, one of the worst parts about Mac OS X software distribution that you’ll ever run into is creating an installer package for Apple’s Installer. It will frustrate the heck out of you, cause you to curse out loud, and generally cause you stress for days at a time until you solve your problem, and then you’ll never want to touch it again for fear of breakage.

In addition, if you’re deploying for previous versions of Mac OS X, you have to live with all the bugs of the previous versions of the installers in order to be able to target them. And that’s where things get really, really icky. You’ll get reports from users about your installer not working, and you will struggle to figure out just what that problem is. This article serves as (what I hope will be) the definitive guide for knowing all the pitfalls and workarounds to the atrocious bugs that exist in Apple’s older installer technology.

I’ve actually had this guide semi-published for a while now, but never got around to finishing it. I’m doing so now. It’s kind of outdated what with the Mac App Store, but if you do ever need to touch PackageMaker and/or Installer, hopefully this guide will help you through the nightmare.

This article will inform you regarding Mac OS X versions 10.4-10.6. It hasn’t been specifically checked for changes on 10.7, but as far as I know, the general situation hasn’t changed with Lion. As for older versions, Panther and previous releases of Mac OS X are dead to me, as they are to the vast majority of Mac OS X users. While there are some users for which these older releases are no doubt more than enough, I doubt it’s worth any effort anymore to target these users.

There are three sections to this guide. The first outlines specific problems that you might want to solve when creating an installer, their solutions, and their pitfalls. The second section outlines specific annoyances with PackageMaker that you need to be aware of. The third section provides a general recommendation of how you should structure your installer.

If you’re using Lion and find any inaccuracies with this guide, please let me know.

Section 1: Installer Problems and Solutions

Problem: You want to install items into the current user’s home folder. Solution: Use the new “domain” feature for 10.5 installers to target the user’s home. Pitfall: FileVault users will be unable to use your installer. OS Compatibility: 10.4: unsupported. 10.5: fatally buggy. 10.6: Works correctly, recommended.

Details: Before Leopard, there was no easy way to install items using Apple’s installer to the user’s home directory. Leopard introduced a “domain” feature that allowed you to target the user’s home directory. If you had a package that installed something to /Library/Internet Plug-ins/ , you could offer the user a choice: install for all users (which installed to /Library/Internet Plug-ins/ ), or install only for the current user (which installed to ~/Library/Internet Plug-ins/ ). This is a great feature, if it only it would work correctly and consistently in all cases.

Unfortunately, as you’ll see is typical of Apple’s installer technology, it doesn’t. When FileVault users try to install to the home directory on Leopard, three things will happen:

  1. The installer will hang for about 30 seconds when attempting to change to the “Select a Destination” screen.
  2. You’ll see this error in the Console:

    Installer[23479]: -[PFReceiptDB initWithPkgID:andName:onVolumeOrHomeDir:] some arguments are required! pkgid: ‘com.packagemaker.simx.testapp.bugDemonstration.test.pkg’ domainPath: ‘(null)’

  3. When the “Select a Destination” screen finally comes up, the user can opt to install to their home folder, but the “Continue” button will be grayed out if they do so. They will be unable to proceed with home folder installation.

Workaround: If your installer does not depend on you installing to their home folder, that’s good news. Just deselect the “home folder” checkbox in PackageMaker, and rebuild your installer.

In many cases, however, it’s better to allow the user to install to their home folder. With ClickToFlash, for example, we don’t want to force our software onto all users of a machine simply because one user installed it. We want one user to be able to use ClickToFlash and allow another to not have it installed at all.

The workaround in this case is to install the necessary files and folders to /tmp , which any user can read and write to. Then, in a postinstall, postupgrade, or postflight script, move the items from /tmp to the correct location in the user’s home folder.

The good news is that this bug is fixed in 10.6. Hooray! If you’re targeting 10.6 only, feel free to use the “domain” feature to install to the user’s home directory.

Problem: You want all users to be able to run your installer, including non-admins. Solution: Do not require an administrator password. Pitfall: Once one user installs your piece of software, other users who attempt to install it will fail. OS Compatibility: 10.4: fatally buggy. 10.5: fatally buggy. 10.6: Works correctly.

Details: Most installers request an administrator password to be able to install your software. However, you may have non-admin users of your software that want to install it as well. If you’re trying to install to the user’s home folder using the workaround method as described above (install to /tmp and then moving in a postflight script), you will want to allow non-admin users to install your software without an administrator password.

The problem is, this doesn’t work so well. In both 10.4 and 10.5, installers leave a receipt of the package behind in ~/Library/Receipts/db/a.receiptdb, and this causes problems when another user tries to install it. Installation by the other user attempts to update this receipt, but if they are not an admin user, they are unable to do so, and installation fails with the following error message:

“installdb[2296]: Error: Could not update existing receipt using uid=503 as the prior owner is uid=501”.

Unfortunately, postflight scripts are run after the receipt is updated, so if you have critical commands in your postflight script, this is a real installer failure.

Workaround: The workaround to this problem is to tell the receipts database to immediately forget that the installer has been run before, by issuing it a forget command. To do so, you need to set up a postflight script, with the line: sudo pkgutil ‐‐forget com.packagemaker.testapp.blah.pkg that runs immediately after the user installs the software. This forgets the package was installed, and allows other users to update the receipts database because they will not need admin privs to overwrite the existing receipt.

Problem: You want all users to be able to run your installer, including non-admins. Solution: Do not require an administrator password. Pitfall: If an admin user has not yet installed the software, non-admin users who try to install it will fail. OS Compatibility: 10.4: fatally buggy. 10.5: fatally buggy. 10.6: Works correctly.

Details: Yes, you read correctly, this is the exact opposite of the above problem. Non-admin users are screwed on 10.4 and 10.5.

In this case, this issue is caused by installer packages writing their list of materials as a receipt to another location on disk, so that when the user repairs permissions, the list of materials will allow restoration of the original permissions as originally installed. These are the traditional receipts that you see in /Library/Receipts.

The problem here is that non-admin users are not allowed to write to /Library/Receipts. So if the software hasn’t been installed yet, the installer run under the non-admin user will attempt to write the receipt and will fail. As with the previous issue, postflight scripts will fail to run, and this is a real installer failure.

Workaround: Fortunately, if the Installer finds any file where the receipt should be, it just throws up its hands and doesn’t even attempt to write the receipt file. So, your job is to trick the Installer that the installation has already happened. To do so, just create a zero-length file with the correct filename. If your package is named “ClickToFlash.pkg” for example, then simply issue the following command in the Terminal: su admin_user -c ‘touch /Library/Receipts/ClickToFlash.pkg’

Unfortunately, this Terminal command still requires admin privileges, and there’s no automated way to work around this problem. You’ll have to tell your users to manually run this Terminal command. Alternatively, you could create a GUI application that runs this command for them and simply requests an administrator password in a nice password dialog using Authorization Services — this requires more work but is more user-friendly.

Problem: You want all users to be able to run your installer, including non-admins, but you’re running up against the previous issues. Solution: Detect whether an administrator password will be required, and ask for if it if it is needed. Pitfall: The installer will occasionally deadlock when you attempt to install. OS Compatibility: 10.4: fatally buggy. 10.5: fatally buggy. 10.6: Unknown.

Details: So, you’re clever and you want to only request an administrator password when it is absolutely necessary, and allow non-admins to install if they are able to do so without an administrator password.

So you set up a meta-package, which contains two installer packages: one requires an admin password for installation, and one doesn’t. They both contain the exact same set of files. You’re even more clever because you symbolic linked the contents of one of the packages to the other, so you’re not actually forcing users to download the same bytes twice in order to install your software. Then, in the requirements check, you run a script that tests for the existence of the receipt, and requests an administrator password if it isn’t there, and allows installation without a password if it already exists.

Yes. I’ve done this for ClickToFlash. And the Installer doesn’t like it. It deadlocks with the following Console messages: com.apple.launchd[1] (com.apple.installdb.system) Throttling respawn: Will start in 3 seconds. And then you’ll see this:

Installer[8601] *** -[NSLock lock]: deadlock ( ‘(null)’)
Installer[8601] *** Break on _NSLockError() to debug.

To the user, what happens is that the Installer will hang at the “Select a Destination” screen, and will show the spinning rainbow cursor of death. The Installer will need to be force quit.

The most frustrating thing about this bug is that it is so rare and yet completely reproducible at the same time. The bug seems to only show up once per computer, usually the very first time they run this type of installer. @rentzsch noticed this as soon as I had created this installer for ClickToFlash.

Workaround: None.

Section 2: PackageMaker Problems and Solutions

Problem: You want installed items from a package for 10.4-10.5 to have their owner be the installing user. Solution: Not obvious. OS Compatibility: 10.4: Problem exists, use workaround. 10.5: Problem exists, use workaround. 10.6: Problem exists, use workaround.

Details: In the current version of PackageMaker, you can explicitly set the permissions on all the items in your package. However, there’s no obvious way to tell PackageMaker to install using the installing user’s user and group. There is no “current user” or “current group” option in the user and group pop-up menus. If you specify your own user and group (that is, the user and group of the developer creating the installer), PackageMaker installs the contents of your package with an owner of “root” and a group of “admin”. This is bad if you’re installing to a user’s home folder.

Workaround: The best workaround is to use a command-line tool to build your installer. Not only will this assist you in automated creation of your installer packages when you update your software, but it allows you to specify no permissions, and thus the installer will install the files using the installing user as the owner.

Here’s the command we used for ClickToFlash:

“$SYSTEM_DEVELOPER_UTILITIES_DIR/PackageMaker.app/Contents/MacOS/PackageMaker” \
‐‐root “$BUILT_PRODUCTS_DIR/ClickToFlash.dst” \
‐‐no-relocate \
‐‐info Info.plist \
‐‐resources resources \
‐‐scripts scripts \
‐‐target 10.4 \
‐‐version “$PRODUCT_VERSION” \
‐‐verbose \
‐‐out “$BUILT_PKG”

The ‐‐root option specifies where intermediate products will be built. The Info.plist file supplied to the ‐‐info option are the options provided to the installer. You can create a package using PackageMaker and build the package, and then just filch the Info.plist file for this command. The ‐‐resources option specifies a folder which contains items for the installer: “background.tiff” to supply the background image for the installer, “.lproj” folders (inside which contain “ReadMe.rtf”, etc.) which specify the language resources for the installer, etc. The ‐‐script option specifies a folder for preflight, postflight, and other installer scripts. The ‐‐target option specifies the minimum system version you want your installer to work on. Provide a product version number to ‐‐version and a path to ‐‐out and PackageMaker will deposit a package at the specified path. ‐‐verbose simply tells the command to show you what it’s doing, and the ‐‐no-relocate option I’ll talk about in a bit.

Problem: You want to install only if an app is a certain version. For example, you want Safari to be version 3.0 or higher. Solution: Add a requirement that checks for that version, by specifying the path of the application. Pitfall: Installer completely fails if the user has moved that application from your specified location. OS Compatibility: 10.4: Problem exists, use workaround. 10.5: Problem exists, use workaround. 10.6: Problem exists, use workaround.

Details: This is an annoying problem. When you want to specify that a certain application is at least a certain version before installing, the only option that PackageMaker allows you to choose is to specify the application’s location by a defined file path.

Unfortunately, there’s a problem that happens if the user doesn’t have that application at the specified path: PackageMaker fails to create JavaScript code that tests for the existence of the file at the path before attempting to ascertain its version. For example, if the user has moved Safari out of the Applications folder, this will cause the installer to fail. It will put up a message saying “There was an error while evaluating JavaScript for the package.”

Workaround: Write this requirement out in red in the read me file for your installer.

You can try to fiddle with the JavaScript using the Raw Editing Mode of PackageMaker, but see the next issue for why you probably shouldn’t do that.

Problem: You want to customize the JavaScript that PackageMaker creates for requirements on installers. Solution: Use PackageMaker’s raw editing mode, of course! Pitfall: It just doesn’t work. Randomly. OS Compatibility: 10.4: Problem exists. 10.5: Problem exists. 10.6: Problem exists.

Details: Whenever I’ve used PackageMaker’s Raw Editing Mode, it simply causes the installer to fail. I was editing whitespace in the JavaScript that PackageMaker had created, and that caused the installer to put up an error message saying, “There was an error while evaluating JavaScript for the package.”

I looked at the raw distribution.dst files that PackageMaker creates for your packages, and the JavaScript looked OK, so that didn’t seem to be the problem. It seemed like the Installer was having trouble evaluating the JavaScript for some reason.

Later, Raw Editing Mode seemed to work OK. This may have been a one-off quirk, but I just can’t rely on any custom JavaScript to work.

Workaround: None known.

Problem: You want to include a helper application in your installer, one that is contained within your larger app’s bundle. Solution: Seems obvious, just include it as one of the files to be installed. Pitfall: It gets “relocated”. That is, your helper application gets installed on the same level as the larger app. It does not stay within the larger app’s bundle, even though the contents clearly show it within the bundle in PackageMaker. OS Compatibility: 10.4: OK. 10.5: OK. 10.6: Problem exists, use workaround.

Details: Your helper app gets installed on the same level as your larger app. This is annoying, because it shouldn’t be there, and it causes your larger app not to know where the helper app is.

Workaround: Create your installer using a command-line tool, and include the ‐‐no-relocate option when creating your package. See the first issue in Section 2 for a sample command that will do this.

Hat Tip: Peter Hosey pointed this option out to me.

Problem: You want to include multiple items to install while using PackageMaker. Solution: Drag and drop them onto the list of contents at the left side of the PackageMaker window. Pitfall: This creates a single installer package for every single item on that list.

Details: This is one of those weird PackageMaker UI quirks. When you drag items to the list of contents in the PackageMaker window, PackageMaker assumes that you want a separate choice for each of those items.

A choice appears when you do a custom install. It allows the user to pick and choose which things he wants to install (and in some cases, where). Usually, though, you just want one choice for your entire installation, and so you want just one package.

Workaround: Create a new choice using the [Project —> Add Choice] menu item. Then drag files you want to install to that choice. This is better than creating a single choice for each item.

Problem: You want to install items to global folders while constructing your installer using PackageMaker. Solution: Change the destination for each item in your choice to the correct install path. Pitfall: Installation fails to install items to the correct locations. Randomly. Installer does not inform the user that the install failed.

Details: This is just annoying. You specify a destination path for your files, and they just don’t get installed.

Workaround: Create a folder for all the global paths, and then install to /. For example, if you wanted to install to the global applications folder, you’d create a folder named “Applications”, add that to your list of contents, and then specify the install location as /. Or, if you want to install to the global internet plug-ins folder, you’d create a folder called “Library”, inside of which you’d put a folder named “Internet Plug-ins”, inside of which you’d put the plug-ins you want to install. Then you’d drag the “Library” folder to your list of contents, and specify the installation destination as /.

There are two very important options that you need to be aware of when using this method. First, you do not want to overwrite the permissions on the global folders. It would be bad if you set the owner of the /Applications/ or /Library/ folders to something other than their defaults.

To make sure this doesn’t happen, double-click on one of the contents in the list, or select one of the contents and choose [Package —> Package Flags…]. Make sure that “Overwrite Directory Permissions” is not checked!

Now, you need to make sure that the installer knows that it’s installing your items inside those global folders. To do so, click on one of the packages in the list of contents, and then click on the “Contents” tab. Here, make sure the “Include root in package” box is checked. In our examples above, this causes the package to “install” a /Applications/ folder, but since it already exists, it doesn’t touch the permissions and installs the files just where you want them.

Problem: You want to specify the destination of a package to be an absolute path or relative to the project. Solution: Select the correct option in the gear menu to the right of the Destination text field in PackageMaker. Pitfall: A UI bug makes it unclear what is actually happening.

Details: Intially, when you create a new package, neither “Absolute” nor “Relative to Project” are checked in the gear menu for the Destination field.

However, if you try to choose either of them, both of them end up being checked.

Here’s what you need to know: when neither option is checked, you’re using an absolute path. When both options are checked, you are using a relative path.

Section 3: General Recommendations for Installers

Given all these problems and issues with the Installer and PackageMaker, it’s worth noting the best way to get around these issues. Here are my recommendations:

  1. Target 10.6 and 10.7 only. If at all possible, TOTALLY DO THIS. You will save yourself so many headaches and problems, trust me.

  2. If you must support versions of Mac OS X below 10.6, I highly recommend creating two different installers: one for 10.6 and above, and one for 10.5 and below. Because of the FileVault installer bug (the first issue in Section 1), I just cannot recommend using the “domain” feature at all on 10.5. You should stick with 10.4 installation mechanisms on 10.5. But you can still reap the benefits of 10.6’s newly (mostly) bug-free installation this way.

  3. For your combined 10.4/10.5 installer (or for combined 10.4-10.6 installers, too), require an administrator password. Just do this, it saves you so many headaches, too. You’ll still be able to install to their home folder if necessary (by installing to /tmp and then moving with a postflight script), but it’ll still require an administrator password. Unfortunately, this is the reality. (For your 10.6 installer, by all means, use the “domain” feature and don’t require an administrator password for installation to the home folder.)


Tips   Permalink   Post a Comment

On the PROTECT IP Act

domenica, 2012-01-22; 23:16:27



On Wednesday, Stop SOPA Day, I took the time to write out two snail mail letters to Senators Barbara Boxer and Dianne Feinstein. I addressed, stamped, and sent them off that evening. Even though both SOPA and the PROTECT IP Act have been delayed indefinitely, the letters will probably have arrived after that announcement by committee chairs. Nevertheless, here is the full text:

Dear Senator [Barbara Boxer or Dianne Feinstein],

I am a resident of California, and I’m writing to protest your pledged support for the PROTECT IP Act, a law that will threaten the very open and innovative nature of the internet.

It is appalling to me that a longtime U.S. Senator such as yourself does not avail herself of the numerous technology experts that almost unanimously say that this legislation is incredibly harmful to the internet. It is appalling to me that the hearings on this legislation were packed with media conglomerates who have a vested interest in protecting an outdated business model, and only one technology company was invited to participate. It is appalling to me that my representative — you — is actively ignoring the possibility of educating herself about technology, and is willfully remaining ignorant because of the money that media companies have contributed to her campaign: [$571,600 or $54,750].

In addition, the supposed impetus behind this legislation is to combat “piracy”, a problem that is almost farcically false and made up. Tim O’Reilly, publisher of technology books at O’Reilly Media and a person who makes money off of creative works, actively questions whether piracy is really a problem. Please read his post here:

https://plus.google.com/107033731246200681024/posts/BEDukdz2B1r

Piracy is not a problem. Existing laws protect creative works well enough, and copyright law is actually overprotective. The duration of copyright, under the Sonny Bono Copyright Term Extension Act of 1998, is lifetime of the author plus 70 years. This is ludicrously long, and prevents even works from the mid-1800s from being released into the public domain, where everyone would ultimately benefit from the wide availability of content. And current law imposes penalties of up to $150,000 per work, which is absurdly out of touch with the actual amount of monetary damage caused.

I am a software engineer, and while I don’t enjoy it when people don’t pay for my work, I am not concerned about the problem. I still get paid without threatening anybody for “stealing” the apps that I write. Have you actually talked to anyone about this issue?

My representative, Rep. Anna Eshoo (and, soon, Rep. Nancy Pelosi) is opposed to the PROTECT IP Act’s counterpart in the House, the Stop Online Piracy Act. Talk to her about this issue, and find out why she opposes it.

For more information, here is a technical analysis of why PROTECT IP would be devastating to the internet:

http://blog.reddit.com/2012/01/technical-examination-of-sopa-and.html

Know that if you vote for the PROTECT IP Act, I will not vote for you in the next election.

Sincerely,

Simone Manganelli

Photo of PROTECT IP Letter

ProPublica publishes total campaign contributions related to SOPA and PROTECT IP.


Publications   Permalink   Post a Comment

On Trolls

venerdì, 2011-08-12; 19:33:57



Me on the App Store, regarding Shaun Inman’s new iOS game The Last Rocket:

This game has a lot of potential. There are cool gameplay mechanics, interesting puzzles, and great 8-bit graphics and music.

Unfortunately, the responsiveness of the touch controls is terrible. The game tends to drop a significant portion of your touches when they are done in quick succession, for no good reason at all. Try it! While airborne, tap at a constant rate of about 6 times a second, and the rocket doesn’t hover in the same place… instead, it kind of schizophrenically moves randomly in either direction, indicating that your touches aren’t being recognized at the proper rate.

This manifests itself in a number of different ways, primarily making it hard to time a quick reverse when there is no room or when you’re heading toward a death. So many times I’ve been rocketing towards some spikes and clearly tap the screen well before the spikes, yet I run into them anyway. Also, if you need to do a quick reverse after turning, forget about it — you basically have to double-tap to ensure the game registers your intentions. Level 6-8 is maddening with this defect, because you have to make quick turnarounds in very little space, and it’s almost impossible to do this correctly.

An otherwise awesome and fun game is made infuriatingly frustrating by this one problem.

For the dev only: simX [at] mac [dawt] com to respond.



Some guy, posting a pic of my review on Twitter:

The guy who called MacHeist customers cheap fucking bastards, turns out he’s a cheap bastard of an App Store customer.

and, after me being pointed out on Twitter:

dear @simx you’re a piece of shit and your controls blow. 1 star as a person.

Then he says:

the guy’s just a troll, he runs his mouth off to get attention. hence him leaving his email address. I guess I’m feeding… it.

and

if you as a customer leave an exaggerated review or rating in an effort to extort changes in a $2 or $3 app you’re a cheap bastard

and

I seem to see most customer reviews raving about the controls in this game, so I must be totally, retardedly wrong.

and a bit later:

everyone rants, not everyone has over 20% of their blog posts filed under the rant category. as I said, it’s his schtick.

and

I’ve looked through @SimX’s App Store reviews and I’ve decided he wasn’t trolling TLR, he’s just a hyper-critical dick. I can be that too.

and

What really got me was his 2/5 star rating equating to an F



So let’s see…

  1. I actually bought the game at $3 and I didn’t mention the price at all. Clearly I’m cheap.

  2. I leave my e-mail address, which is how he managed to connect my App Store review and my weblog post, but that somehow makes me a troll, instead of someone not wanting to leave an anonymous review. (Amusingly, he couldn’t even find me on Twitter before having it pointed out to him, despite my nickname on the App Store being exactly the same as mine on Twitter.)

  3. The app has 71 5-star ratings, and mine is one of three 1- or 2-star ratings. Clearly my review was to extort changes out of Shaun Inman, because I’m totally hurting his sales.

  4. Everybody else says something isn’t wrong, so it must be so.

  5. It’s someone’s “shtick” if they do it 20% of the time. Not the majority of the time, just 20%.

  6. I left a detailed review outlining my one issue with the game, and have done so with other apps as well, but that classifies me as a “dick”.

  7. 2-star ratings are classified as F’s. Assuming he’s using the American system where anything less than 60% is an F, then 3-star ratings barely qualify as D-minuses. And 4-star ratings qualify as B-minuses. That makes so much more sense than 1-star being “hate it” and 2-star being “dislike it”.

By the way:

My awards in The Last Rocket

I’ve played it through four times. I stand by my review.


Intarweb   Permalink   Post a Comment

On Ambrosia Software's Tweetspam

lunedì, 2010-04-05; 13:56:46



Last July, I interviewed Andrew Welch, President of Ambrosia Software, Inc., regarding the “Tweetblast” that MacHeist used to promote its software bundle. Twitter users would post a specifically crafted tweet to their followers, in exchange for some free software from MacHeist. This is what Welch had to say:

I do think that the Tweetblast was somewhat of an abuse of the power of Twitter.

[…]

If you want to ask me the real question, that is, would Ambrosia do something like this on our own to promote our products, the answer is no. We would not. I’m not sure if that’s good or bad.

Fast forward to today, about eight months later, here’s Ambrosia’s website:

Ambrosia Tweetblast Promotion

This makes me angry for a number of reasons.

First, and most obviously, Ambrosia’s new promotion is a direct contradiction of what Welch said eight months ago. I realize that there was no promise that they wouldn’t re-evaluate their priorities and do it in the future, nor was there a specific timeframe qualifier. It’s Ambrosia’s prerogative to promote their products however they want. But it’s my prerogative to be annoyed at sleazy, spammy marketing tactics.

Second and more importantly, though, is that Ambrosia learned absolutely nothing about the MacHeist Tweetblast promotion, and Welch in particular seemed to learn absolutely nothing from my interview with him. Here’s what I said then:

Well, in any case, that situation is also different because it’s someone organically telling someone about another product. It’s not Mary Kay directing to this person exactly how and when to say it, and offering goods in return.

I mean, I would be far more charitable to the TweetBlast if, at the very least, there was a way for them to accept any form of tweet as long as it mentioned “MacHeist” and “Delicious Library 2”, and not a specifically crafted tweet.

Go back to Ambrosia’s promo page for the Tweetblast. You first have to follow @AmbrosiaSW on Twitter. Then, if you click on the link in Step 2, you are taken to the Twitter homepage where a specifically crafted tweet is entered for you on your timeline.

*facepalm*

Welch said that this is no different from a friend periodically asking him if he is interested in Mary Kay products. I beg to differ. This is like having Mary Kay tell dozens of your friends to ask you all at once, using Mary Kay’s specifically-crafted promotion language, to go buy some of their products. It’s so fucking obnoxious.

But here’s the real kicker: Ambrosia actively flouts Twitter’s efforts to combat this type of spam.

Back in November, Twitter introduced their own native retweet feature. While I despise retweeting for the exact same reasons that I despise Tweetspamming, it does have one redeeming aspect: if a bunch of people you follow on Twitter retweet the same tweet, it only shows up once in your timeline. That’s awesome. At maximum, I’ll get a single spammy tweet for a promotion if people use the retweet feature.

But no, what does Ambrosia’s website ask you to do? It gives you a link that does not use the retweet feature. Basically, Ambrosia is asking you to get around a really nice antispam feature of Twitter. If you go to Ambrosia’s Twitter account page, there is no tweet advertising the promotion that you can retweet, and still be entered into the contest to get the iPad.

Why is Ambrosia doing this? Simple. Because it gets more eyeballs to their pages. The more times you see tweetspam in your timeline that advertises Ambrosia’s products, the more sales they get. It’s sleazy marketing at its best.

Dan Wood asks on Twitter if there’s “some approach to this general idea that wouldn’t be so annoying”. Yes, there is. I had already explained this to Welch directly, but apparently it didn’t sink in, so I guess I have to repeat myself.

Here’s how you can do an effective, non-annoying tweetblast promotion:

  1. FUCKING USE THE FUCKING SEARCH APIs. Seriously, the search APIs for Twitter have been around for over a year. Let people craft a tweet in whatever way they want, as long as they include specific words. In Ambrosia’s case, they can mandate that the words “@AmbrosiaSW” and “iPad giveaway” are mentioned in a tweet. If they are, you’re entered in the contest. Yes, this means that you’ll have to allow entries that say “@AmbrosiaSW’s Tweetblast for their iPad giveaway is fucking obnoxious”, but them’s the fucking breaks when you want to do non-invasive marketing.

  2. DO NOT GIVE OUT A LINK THAT FILLS IN PEOPLE’S TWITTER STATUS FOR THEM. Don’t fucking do it. Just don’t. If people really want that fucking iPad, they can go and exercise their little spammy fingers for 140 characters in their own way. Filling in people’s status for them just encourages laziness: people will just tweet what you put in for them.

  3. ENCOURAGE USE OF THE RETWEET FEATURE. Allow entries that are retweets of someone else’s valid entries to the contest. That way, people who don’t give a flying fuck about your iPad promotion will only see the spam once.

To those who feel the need to spam Twitter in order to get an iPad or whatever the next tweetspam-du-jour prize is, I recommend this: make a dedicated Twitter account for tweetspam. Who cares if no one is following that account? You’ll satisfy the contest requirements, you’ll get your shot at whatever it is that turns you into a sleazy pawn of a company’s marketing department, and you won’t annoy anybody. KTHXBAI!

Moral of the story: even a seemingly-upstanding company like Ambrosia eventually sinks down to sleazy tactics, and the credibility of Andrew Welch has been damaged. I am seriously re-evaluating whether I will buy any Ambrosia products in the future.


Rants   Permalink   Post a Comment

Finding the Right Cell Phone Plan

giovedì, 2009-10-22; 16:54:03



People are sometimes surprised to find out that I, despite being a computer programmer and keeping up with much of the Mac and technology news scene, don’t actually have a cell phone.

There’s are two reasons for this: 1) I don’t want to be on call all the time, and 2) I don’t want to spend a lot of money. Reason #1 is frequently misunderstood: yes I know that I can simply turn off the phone. Duh. But what’s the point? If I get a cell phone and have it off all the time, it’s just as good as not having one at all. Besides, even if I did, I would be tempted to have it on all the time, and that’s the problem for me. I don’t trust myself not to answer the phone when I shouldn’t.

(For what it’s worth, I hate it when people interrupt an in-person conversation to answer the phone. I also hate it when people make a call in the car, or in a closed place where other people are forced to listen to your conversation and can’t have one of their own. It’s obnoxious. Stop it.)

Reason #2 is an easier point to get across: how much are you all paying for your iPhone plans again? (I think my bro pays almost $100/month.) I’m not willing to throw that much cash at AT&T.

My current setup is a 2-year-old MacBook, and a Skype subscription. For 45 euros (that’s ~$68) per year, I get unlimited phone calls to the U.S. & Canada and voice mail.

Per year.

Oftentimes I lug my laptop around just to be able to have Skype access, though. And that also requires that there’s a wireless hot spot I can access, or have access to a friend’s wireless network. So it’s definitely inconvenient at times.

I’m looking to maybe get a cell phone if it can fit my requirements. Here they are:

  1. The “phone” must be an iPhone or iPod touch. I’m going to be doing iPhone development, and besides, I’ve used all the other phones a bit and I’ve read about them, and they’re just not going to work out for me. So just don’t bother with even mentioning any other device. Seriously, just don’t.

  2. I want a persistent, unlimited wireless data connection, available when I need it, with reasonable coverage. Achieving this must not require a laptop.

  3. I will pay no more than $30/month.

  4. I do not care about phone “minutes”, and I do not give a shit about SMS or MMS (seriously, why do you guys pay for that crap when you can just use AIM?). I do not want them, nor will I pay for them. I will continue to use Skype.

  5. I am willing to entertain hacks or add-ons that can help fulfill the previous requirements (i.e.: if there’s a dongle that acts as a cell phone modem and doesn’t cost much per month, that is acceptable). I am not willing to endure hacks that continually break or are unstable, etc. It can have an up-front time and/or money commitment (up to $200), but this must be a one-time sunk cost only.

  6. I must be able to upgrade to the latest software update as soon as it comes out. No delays. (This pretty much rules out any jailbreaking/unlocking of any kind.)

  7. I need at least 40 GB of space. My music collection is already 30 GB in size, and I’m not picking and choosing what music I want on my device.

Here’s what I’ve looked into:

  1. My bro and dad have a family plan. I could attach on to that, but that costs $10/month just to add another device to the plan, and another $30/month for the data. That violates requirement #3. Also, that would require an iPhone, which, at max, goes to 32 GB of storage, violating requirement #7.

  2. AT&T has a pay-as-you-go plan. I would never use the minutes so I would never need to refill. However, it seems that using such a plan, the data is also pay-as-you-go. That’s going to violate requirement #3 as well. Also, still the problem with not enough space.

  3. Verizon has a dongle-thingy (MiFi) that hooks into its wireless network and provides a roaming wireless hot spot, but last I checked this was way too expensive. It’s $40/month for a piddly 250 MB/month, and 10¢ per MB above that. Hah! Sorry, but no.

Any other suggestions?


Tips   Permalink   Post a Comment

ClickToFlash Killers

venerdì, 2009-10-02; 02:38:49



One of the greatest features of ClickToFlash, in my opinion, is the ability to load H.264 files from YouTube in QuickTime, rather than having to load the Flash plugin. Just searching for ClickToFlash on Twitter reveals a number of users who absolutely love the feature. It’s amazing what this one feature can do to your CPU load, fan volume, and battery life, just by loading videos in a non-horribly-written plug-in. You might even say that ClickToFlash has significantly contributed to the fight against global warming!

If you happen to be an Adobe Flash plug-in developer and you’re reading this, seriously, WHAT THE FUCK? The Flash plug-in has been on the Mac for around a decade, and we’ve been promised better performance time and time again. Yet, still, the Mac Flash plug-in uses around 75% of the CPU under normal circumstances, compared to 10-25% of the CPU that the QuickTime plug-in typically uses and the good performance of the Windows Flash plug-in. The only logical conclusion is that your code SUCKS ASS.

The problem is, the code to load the YouTube H.264 files is entirely custom. What if we wanted to do the same thing for a different video site, like, say, Vimeo? We’d have to write completely new code to account for this.

Well, I want to tackle this problem sooner rather than later. Eliminating more bits of Flash from the web is cathartic. There has been sparse talk about how to implement this (see the ticket for the feature request), but no concrete plans. (@atnan has already figured out a general method for getting H.264 videos from Vimeo.)

I’ve been thinking about this problem for a while. But before figuring out an implementation proposal, we have to think about the goals of such a feature:

  1. Easily-supported code. This means that we need to have all site plug-ins go through the exact same mechanism. This includes the current custom YouTube support. We want only one mechanism to maintain. If there are bugs, we can fix it in one place, and all site plug-ins will reap the fix.

  2. User-extensible. We want users to be able to create their own site plug-ins and share them with the world, without having to learn Objective-C or go through the horrible UI that is git. Part of this means site plug-ins that are relatively easy to create, with little programming knowledge, if possible.

  3. Flexible. Video sites will do weird things. With YouTube, we already have to do HTML scraping for embedded videos. And we can’t rely on the flash variables that YouTube videos include, because sometimes they indicate the existence of a H.264 file that just isn’t there! So we have to test for file existence at a certain URL. In short, the mechanism needs to be able to support whatever video site users think of throwing at it. If we need to add a feature, we want to be able to do so without breaking existing site plug-ins.

  4. Secure. One possible example of an implementation is simply to have site plug-ins be executables which return a string for the URL of the video file. But think about how much of a security hole this is: the executable could happily delete your home folder! We don’t want that possibility with our site plug-ins. That can conflict with our goal of flexibility, but we want to achieve a happy medium between the two.

Today, on Twitter, I mentioned that I wanted to get this feature moving, and @boredzo came up with a preliminary implementation: define site plug-ins as a dictionary file. His proposal lacked some flexibility even for YouTube, but it was a good jumping-off point. I spent a few hours looking through ClickToFlash’s YouTube code and created a dictionary file format that I think can reasonably satisfy all of the above four goals.

Because “site plug-ins” sounds boring, and ClickToFlash already is a “plug-in”, having plug-ins for a plug-in is kind of weird. The name “extension” is already used by Firefox (let’s save a rant about Firefox for another day entirely). So I think we need a new name for these site plug-ins.

In the spirit of the cheeky copy on the ClickToFlash website, I’d like to dub this feature “Flash killers”, or “killers” for short. Yeah, yeah, I know, some of you will disapprove, but it’s one of the little things I do to keep me sane as a programmer.

Anyway, I’d like to walk you through three sample killers: two that replicate the existing functionality of ClickToFlash for YouTube, and one that would introduce H.264 support for Vimeo. Note that not a single line of Objective-C code has been written to support killers, at this moment in time. The whole reason for this post is to receive constructive criticism on the proposal before implementation.

For that reason, I ask that you read this entire post before commenting. I’ll explain the decisions behind everything.

As mentioned before, killers are simply .plist files, or a dictionary of objects and keys that define how ClickToFlash will construct the URL for a video file. Here’s the root view of a killer for YouTube:

“Killer Name” and “Video Result MIME Type” are pretty self-explanatory.

Here’s the same killer showing the contents of the “Matching Regular Expression” and “Rejecting Regular Expression” dictionaries:

There are three things that we use in the current ClickToFlash code to check for YouTube videos: the URL of the page that you’re on (the one you see in Safari’s address bar), the URL of the source flash file (the “.swf” file), and the flash variables themselves (which can contain the URL). For embedded YouTube videos, we need to support regex matching of each of these URLs. This killer does not support embedded YouTube videos; I’ll explain more about that later. This one just supports actual YouTube pages.

So for non-embedded YouTube, we just need to match “youtube.com” or “youtube-nocookie.com” at the start of the page URL with an optional “http://” and an optional “www.” (Please note, I am a regular expression rookie, so if you see any mistakes, please let me know.)

The “Rejecting Regular Expression” is the opposite — if the URL matches the regular expression in this dictionary, ClickToFlash won’t attempt to load the video file via this killer. Because standard YouTube pages will also match the regular expression for source URL and Flashvars, which we use to check for embedded YouTube videos, we don’t want two killers matching for the same video. (It actually probably doesn’t make that big of a difference if two killers match a given video, as the first killer to be loaded will just take precedence.)

Here we see support for the variants of YouTube. Remember, YouTube has standard H.264 and HD H.264 versions available. We want to support both, and add support for additional variants from other sites without having to create additional killers (which would cause problems, because the regular expression matching for the killers would be identical, at least in the case of YouTube).

Here, “Display Name” specifies what text will be displayed in the ClickToFlash view if the URL successfully loads. “Return URL” also specifies the steps to construct the URL and the return URL string (the format of which we’ll get to in a moment.)

Here we can see how we construct the URL of the MP4 file for YouTube. ClickToFlash will support a limited number of step “types”, including flash-var-retrieve, regular-expression-match, download-resource, and because of this format, we can add support for additional step types in the future if deemed necessary.

Each step can contain three objects: “Type”, “Victim”, and “Action”. Type indicates what ClickToFlash will do in that step, “Victim” indicates the string to perform the step on, and action represents the specific form of the type of step. For flash variable retrieval, we don’t need an action, because this step simply retrieves the string for a given flash variable (specified in “Victim”). Other steps do require an action: for example, a regular expression match actually needs a regular expression.

In addition, each step always returns a string. If a certain step returns nil, then we’ll cancel out of the killer. The result of each step is stored in an array for later retrieval. So in these two steps, we retrieve the video_id and t flash vars.

In the “Return URL” object, we see how we refer to the results of the previous steps. “%3A” and “%4A” represent the results of the steps in the killer. Why start with “%3”? Because “%0” is reserved for the page URL, “%1” is reserved for the Flash source URL, and “%2” is reserved for the Flashvars string. The trailing character of the entity used for retrieving the results of a single step is to allow you to retrieve results from a different variant, which we see will come handy in a sec.

In this pic, you can see that the YouTube HD variant has no steps. How can that be?! Well, we already retrieved the required strings in the previous variant, so we just refer to them again in the return URL. “%3A” and “%4A” will return the same video_id and t flash vars as it did in the previous variant. If we had a third variant, we could refer to the steps of the previous variant using “%3B” and “%4B” instead.

Why support this? Because it saves processing time, and it will save network time in the case that you need to download an external resource, and need to refer to the same resource in two different variants. You don’t want to download the same HTML file twice to scrape (as we might do for this killer), and you don’t want to spend the CPU time to scrape it twice, either.

Now let’s move on to embedded YouTube. Remember how I said we didn’t want proper YouTube page videos to match for two different killers? Here we use the “Rejecting Regular Expression” to that effect. It’ll match “youtube.com” or “youtube-cookie.com”, but if those strings appear in the actual webpage URL (as it does for standard YouTube videos), then this killer will reject the match.

Here’s how we actually construct the URL. Remember when I said we had to do HTML scraping? Here’s where we do it. Our first step is a regular-expression-match on the source URL of the Flash file, to extract the video_id parameter. Then we actually have to download the HTML source of the corresponding YouTube page, which occurs in the second step. In the third step, we do a regular expression match against the HTML source of the previous step, searching for the flash arguments. The last step searches the results of the previous regular expression result to match the “t” flash var, which is the last item we need to construct the URL.

Note that victims do not need to be single entities, which are the results of previous steps. You can construct strings that include the previous step result as a subset, if necessary.

Also, note the regular expression matching here. (?< ) is a “look behind” parameter, forcing a string to have a certain other regular expression come immediately before the string. Similarly, (?= ) is a “look ahead” parameter.

How does the HD variant for embedded YouTube videos work? Easy-peasy! We’ve already retrieved the “video_id” and “t” flash vars from the previous variant! We don’t need to do any work here. Just construct the return URL using the results of the previous variant (which in this case uses “fmt=22” to get the HD version rather than “fmt=18”)!

There’s one more thing you need to be aware of: in ClickToFlash’s current code, we need to actually check for the existence of a file at the constructed URL in order to make sure that one exists. YouTube sometimes lies, and gives you “video_id” and “t” flash vars even though there’s no H.264 or HD version. But this can be handled in ClickToFlash code — ClickToFlash will just check for the existence of a file at the return URL for you!

I hope you agree, this step-by-step method for constructing URLs is sufficiently powerful to allow for flexibility for the various websites out there. Because the number of steps isn’t limited, if you need to download two external resource files, that’s absolutely supported! If you need to do a vast number of regular expression matches and not just one, that’s supported! And given the flexibility of the “Type” parameter, we can support future step types, like CSS selectors, XPath queries, and possibly even AppleScripts!

How does this live up to our original four goals?

  1. Easily maintainable code. A defined set of steps and matching regular expression allows us to write one Objective-C class for every killer.

  2. User-extensiblity is pretty evident. Also, I believe the step-by-step mechanism makes it easier for novices to understand how a killer is set up, and attempt to make one of their own if they so please.

  3. Flexibility is also pretty evident: this mechanism has been designed from the ground-up to support all the quirks of YouTube, and allows for future-proofing by adding additional step types, so I’m confident that it can handle the majority of the video sites out there.

  4. Security: with this proposal, all that a malicious user can do is fetch flash variables, download resources, and match regular expressions on pieces of text. I think that pretty much avoids any potential security holes that such a plug-in mechanism may have. The only exception would be AppleScripts, since they could delete files if they so wanted, but we can discuss whether we want to add support for those or not.

What are the deficiencies in this proposal? I can see a number of them.

  1. It’s verbose. I did this on purpose to aid “readability” of the killers, but I can see how it might be annoying to set up.

  2. Kind of related to the previous deficiency, the mechanism for referring to results of previous steps is kind of awkward. What if someone had a large number of steps? Or what if they wanted to use a victim such as “http://asdf.com/?%4A=something&%3A=something-else”, and not have the “%4A” match the result? Would escaping the entity work reliably?

  3. Localizability. How to localize the keys in the dictionary for other languages, so that non-English speakers can still create killers and get them to run under any system language, including English? Not sure how easy it will be with this mechanism.

I’m not sure the answers to these questions, and maybe they’re things that we don’t really need to worry about. In any case, if you’d like to play around with my Dictionary files, you can access them here:

I’ve even added one that should support Vimeo’s H.264 files!

And now that you’ve read through the whole thing, comment away! I welcome any and all discussion, since this is probably the single biggest feature I’ve attempted to undertake in ClickToFlash, and I want to get it right the first time.


Software Development   Permalink   Post a Comment
Browse the archives