NSLog();

31 Comments »

31 Responses to "Deatherage Drivel on X’s “Open Formats”"

1. | Reply Buzz Andersen
   Posted 08 Apr 2003 at 4:16pm #
   

   Just a small point of clarification: the file format iTunes uses directly is actually a proprietary, binary database file labeled "iTunes 3 Music Library." The XML file is, indeed, generated by iTunes for your convenience, but changes made to the XML will have no effect on the actual library as displayed in iTunes. So apps that want to do anything with the iTunes media library besides display it are SOL.

   The database file on the iPod is similar. I know all of this because I spent a good chunk of time developing an OS X framework that is capable of parsing these database formats. It is definitely not an open standand.

2. | Reply Michael McCracken
   Posted 08 Apr 2003 at 5:05pm #
   

   On a related note, the iChat saved chat files are opaque for no good reason. Check out ~/Documents/iChats/ - why not use XML here too?

3. | Reply Erik J. Barzeski
   Posted 08 Apr 2003 at 5:51pm #
   

   I realize that the databases themselves are not the XML documents. However, Deatherage's assertions that third parties can not get at the data is incorrect and shameful. He said that apps can not get at the information. It's not about changing the database - you can't modify your iTunes library in iPhoto - it's about reading it.

4. | Reply Matt Deatherage
   Posted 08 Apr 2003 at 6:44pm #
   

   Well, it's about a lot of those things, but Erik seems far more enamored of assertions like "crack-smoking" and "what-the-fuck-ville" than advancing the debate. Let's see if we can clear up his misreading of the column's plain text.

   First, any belief that XML is "self-documenting" is so deluded as to barely deserve response. It's like Keynote's XML format: you can see what Keynote puts in one of those files, but there's no guarantee what Keynote will do with slightly different elements or attributes. The APXL Schema is documented in Technical Note #2067, and even though it is twenty-five pages long (not counting the links to other versions), Apple says it is "a companion to the four-part Technical note 'Deconstructing a Keynote Document' which describes the document format for Keynote in-depth."

   Being able to duplicate an existing file without using any other features you may or may not know exists is not "documented," although it is somewhat transparent. The point of documenting a file format is to let anyone use it, not just to let anyone understand what already-created documents mean.

   This is not a minority opinion, either: MacInTouch readers recently linked to reports like this asserting the exact same point about Microsoft Office 2003's "open" XML format: it's not "open" at all because it's not documented. That comes from the OpenOffice.org people, too. I think the language that a "schema" should come from users and not an application is a bit confusing, but the point maintains: without some clue as to how the XML data is to be used, it's a closed ASCII format instead of a closed binary format.

   That's the point with iTunes. Yes, iTunes 3 creates the "iTunes Music Library.xml" file, but it is neither documented nor supported. (Even the name, I suspect, is localized on non-English systems.) Nor is it always found in "~/Music" as Erik asserts; it's kept with the iTunes library. My library is not in the "~/Music" folder, and neither is the XML file. The only mention of the file on Apple's developer site is on a page that clearly does not document the file format, but uses it as an example for working with XML in PHP.

   So what happens if iTunes 3.0.3 changes what the tags mean? Or adds a new element or attribute that turns out to be important? Everyone misses it because no one outside of Apple knows what the file should contain. There's no DTD and no Schema; no one knows what values are legal for what tags, though everyone has a common sense idea. Common sense ideas create bugs when you write code. I suspect but can't prove that the other iLife applications are reading this XML file to get their information, so if Apple updates the XML, it must also update iPhoto, iMovie, and iDVD. That's pretty much the nature of a proprietary file format, no matter how easy you find it to read or slightly modify.

   Any developer who reads the iTunes XML file has taken on the responsibility to update his code when Apple changes the format. Being in English does not make the format "open" or "documented," Erik, no matter how much sense it makes to you. Sorry. So yes, it is that Toast can't read the format. Companies like Roxio don't want to answer thousands of tech support calls and E-mails if iTunes 3.0.3 makes a minor XML change that breaks the interchange. Maybe your shareware author friends can fix their code for such changes overnight; I probably could if I were still writing code. Big companies have slower releases.

   I notice you don't mention iPhoto libraries at all, or any programs reading anything out of them (at least, before yesterday's release of TiVo Desktop 1.0).

   So, Erik, where is the documentation on iPhoto libraries, or is everything you want to be open "self-documenting," too? Where's the documentation on how programs other than Mail can see if a contact is logged on via iChat? Hint: we're looking for developer documentation that can be used in commercial, supportable code. You obviously weren't looking there, or you would have found Apple's clear insistence that only Apple use menu extras, or "menu bar status items."

   This was reported when Jaguar first came out, and is documented by Unsanity in the "read me" file for Menu Extra Enabler, from August 2, 2002. I read this in my copy of TextEdit; perhaps you threw yours away.

   Dismissing AppleWorks as by "the old Apple" is disingenuous, too: Apple has updated it to make it fully Mac OS X native (and that wasn't "13 months before Mac OS X 10.0 shipped"), and it ships on every consumer and education machine the company sells as the primary productivity application. And Erik ignores my other point: AppleWorks 5 file formats were available to developers, at least on some terms. AppleWorks 6 file formats are not, and that's all the "new Apple's" doing.

   So, in short, XML is not self-documenting for any real supported uses, AppleWorks 6 was revised heavily after the date you mention, the XML file formats you mentioned are not documented at all (when they're even found where you say they are), Apple does tell developers not to use menu extras, and the only file formats Apple touts as "open" are the ones for the programs that don't lead the market -- the entire point of the column in the first place, something you fail to tell your readers. When the column goes online, they can make up their own minds about it; it's also still on newsstands for another couple of weeks.

   Oh, and if you hate this column, you'll really hate the June 2003 one. Better stock up on tranquilizers. 🙂

5. | Reply Erik J. Barzeski
   Posted 08 Apr 2003 at 7:24pm #
   

   "First, any belief that XML is "self-documenting" is so deluded as to barely deserve response." Apparently, "<key>Genre</key>" is too tough for you? I don't care about Keynote. Yes, you can keep returning to Keynote, but I'm discussing iLife's apps. Furthermore, Keynote is well documented, and third parties have deciphered the XML and released products that work with it.

   You say "it's not a minority opinion" but that's a moot point: you say many things that are flat out wrong. Opinion matters not when you say "only Apple can read the iTunes Library" - when you are wrong.

   So what happens if iTUnes 3.0.3 changes what the tags mean? Right. Because "Genre" is so likely to change. Get a grip Matt. Could there be a DTD or Schema? Yep. But the fact remains that you are wrong: you say no other apps "can" read use the information. They can. Your continued insistence that Toast "can't" is absurd and the foundation on which you base this is neither common nor correct.

   Then you go on to say this: "I notice you don't mention iPhoto libraries at all, or any programs reading anything out of them (at least, before yesterday's release of TiVo Desktop 1.0)." You apparently didn't read very well. I mentioned Lightbox, from Josh Anon, as having support for iPhoto's photo libraries. I've also build some things which use the iPhoto libraries, and I saw a freeware status menu thing that did so yesterday too.

   I could also add the Safari bookmarks file - nice, open XML. Yummy. Should Apple provide developer documentation for that? No, Matt - it's common sense. You look at the file, you figure it out. So difficult. The man-hours Apple would spend documenting every nook and cranny is time better spent - they must feel - doing other things.

   As to your discussion of menu extras - got a point? You didn't discover anything new. I'v got apps that use NSStatusBars - a method very well supported and encouraged by Apple for third party apps that want a menu on the right side of the menu bar. Apple reserves actual "menu extras" for itself for various reasons, some of which I listed.

   "Dismissing AppleWorks as by 'the old Apple' is disingenuous?" Hardly. AppleWorks 6.0 was released months before even Mac OS X Public Beta. The current version (6.2.4) runs on Mac OS 9 too. It's been updated for Mac OS X, but it hasn't seen a Mac OS X-only release yet. You'll note that all of the apps making heavy use of XML for their information-sharing are X-only. I wouldn't expect Apple to re-engineer AppleWorks' entire document architecture just for X for a x.x.x release. Duh. 😛 AppleWorks' closed file format(s) are the "old Apple" all the way. If they're still closed when we see 7.0, I'll give you a point. Until then, suck on the crack pipe some more.

   If you're going to continue writing a column called "The Big Picture" perhaps you should stop missing it so blatantly.

   P.S. At the top of my iTunes and iPhoto XML files:

   <!DOCTYPE plist PUBLIC "-//Apple Computer//DTD PLIST 1.0//EN" "http://www.apple.com/DTDs/PropertyList-1.0.dtd">

   Looks like a DTD to me.

6. | Reply Steven Canfield
   Posted 08 Apr 2003 at 7:46pm #
   

   A few things I'd like to point out:

   1.) XML is human readable. I can open the iTunes Music Library.xml file, and INSTANTLY I know the format of that file. Using NSDictionary, I can write a program to display that information in like 3 hours (and I'm fairly novice). Compare that to before, without XML. How often do you see old reverse engineered file specs start off with "This first int has to be 9202 or the file won't work. Don't ask us why, it is just that way."

   2.) Lets say I "reverse engineered" the iTunes XML file (reverse engineered used extremely lightly). Now Apple changes it so that instead of song title first, you put album title. With a regular datafile, I am back to square one. With XML, I can SEE this change very easily. Very easily.

   3.) Menu bar items are obscenely easy. Check out this article at CocoaDevCentral. No, they aren't exactly the same as the Apple brand, but they look pretty dang close to correct (I haven't noticed any differences).

   All OS X apps should use property lists. It is ten times easier on the programmer, and five times easier on the user (both figures entirely made up). Your article contends that Apple uses openness as a feature when they've got it and are strongly against it elsewhere. If you want to keep me from reverse engineering your file format, do you export it as XML? Of course not.

7. | Reply Spike
   Posted 08 Apr 2003 at 8:21pm #
   

   Jeez Louise, guys, give it a rest.

   Mac people...

8. | Reply Buzz Andersen
   Posted 08 Apr 2003 at 9:33pm #
   

   Steven does raise a good point--most of the XML used Apple's applications is not just garden variety XML, but rather property list XML, which can be automatically "deserialized" into a Cocoa NSDictionary object. Thus the basic XML structure of the documents is well-defined and not likely to change (it's actually part of Cocoa).

   What *could* change, as Matt points out, is the actual semantic (as opposed to syntatic) aspects of the documents. For example, Apple could decide to abolish the "Track ID" key from the iTunes plist and name it something else, which would in turn break dependent applications. One thing that should make applications less vunerable to this sort of thing, however, is that Apple includes the version number of the file format at the top of the XML, which would enable developers to do what Panic does with Candybar (i.e. produce an error or warning if it encounters an unknown version). I know this is not as good as ironclad documentation from Apple, but it does speak to a certain desire on Apple's part to make the iTunes DB available to third-party developers.

   Of course, I also agree with Matt and Michael Tsai (whose comment has evidently been deleted) when they say that an ironclad, well-documented API for accessing the various iLife media databases would be ideal. I don't think Apple is totally a closed shop, but I don't think they're as open as they could be yet, either.

   P.S.: As for Spike's comment: I understand what you're saying, but, given the vehemence of Erik's, shall we say, criticism, I don't think Matt's response is unfathomable.

9. | Reply Buzz Andersen
   Posted 08 Apr 2003 at 9:36pm #
   

   Sorry, I meant "syntactic" above. My spelling seems to be terrible these days when typing comment posts...

10. | Reply Matt Deatherage
    Posted 08 Apr 2003 at 9:52pm #
    

    The key difference here seems to be the definition of "can." Erik seems to say that if there's no technological barrier to reading the file, then it "can" be read and used. But by that logic, anything "can" be done -- you "can" replace the kernel, reengineer kernel extensions, and parse any data you want.

    But it's not supported. Apple can change it at any time and break your code. The smug assertion that the word "genre" will not change does not guarantee that the element will not change, and most professional programmers don't want to deal with the technical support headaches if it does. iTunes isn't even documented or guaranteed to maintain that XML file at all; what if version 4 drops it for performance reasons and other iLife apps use a different method? Then what?

    When I talk about what programmers "can" do, I mean what's supported, not what can be done with reverse engineering. Anything can be done, as long as you're willing to update it for every release of the OS. RAM Doubler, anyone? TechTool Pro? Now Menus? Sometimes the functionality is worth the risk, and it's a small risk for a shareware programmer. Not so for big companies.

    I spent a lot of years in Apple's DTS group listening to complaints from programmers who had used undocumented file formats, low-memory globals, LaserWriter PostScript dictionary entries, you name it -- all of which eventually broke when system software was revised, and the answer was always the same: Apple never promised those things would work, and you used them at your own risk. Your argument is basically that everyone should be willing to take that risk. The big picture is that not everyone is a shareware programmer who can turn around a new release if a file format changes.

    Look at that iTunes in PHP article I referenced earlier. It outlines a lot of suggestions how the iTunes XML file could be clearer and simpler. What if Apple implements those changes? What if entire elements change for better flexibility? Your code will break, that' swhat will happen. Are you then going to argue that Apple shouldn't be able to improve an undocumented file format because you had chosen to rely on it? This is where a lot of those "the latest Mac OS update broke my program, don't install it" stories come from, and if you want to perpetuate them, don't drag my name into it. If you want to claim that the big picture is that people should unreasonably rely on the format of any given XML file not changing, you're welcome to do so, but you'll deservedly be laughed off the stage. What you call "common sense," professionals rightly called "bad assumptions."

    If AppleWorks 6 is "the old Apple," then why has Apple continually refused to make AppleWorks 6 file formats available under the same terms as AppleWorks 5 file formats, or at least under any terms? Why is Keynote the only program that can read AppleWorks presentations, other than AW itself? Why do file translation tools tell you to convert to AppleWorks 5 format and then import that? By Erik's logic, the long time since AppleWorks 6's release (since he refuses to even consider that the Mac OS X work is more recent than February 2000 despite the plain evidence) should have allowed Apple to make these formats available, even though they're binary and not XML. The simple fact is that Apple locks this information away and refuses to let anyone read or write AppleWorks 6-only files, the files for the only consumer-level integrated software suite of any importance on the platform. Why? Because if you could read and write the files, you could compete with AppleWorks. Apple desperately does not want that, though I think AW could stand against competition quite nicely.

    There's no reason that all of the XML-heavy applications are Mac OS X only, and some are not, though all of Apple's are. The XML routines are in Core Foundation, accessible through Carbon and therefore under Mac OS 9 as well. It's just that most Carbon applications are based on legacy code and have chosen not to rewrite their file code to use XML so far. Arguing that this is why Apple hasn't made AppleWorks file formats available is, well, silly.

    And next time you point to the property list DTD, try looking at it first. It's the DTD for property lists -- the "array", "key", "dict", "string", "date", "data", "true", "false", "real", and "integer" elements. Nothing about anything specific to any program; it's not documentation of any of the file formats you argue are "self-documenting."

    Steven: The argument over whether to use NSStatusItem instead of the Menu Extra API misses the point -- the point was that Apple tells developers not to make "menu bar status items" of any kind in applications, but does it for its own iChat and iSync applications. No one said they were difficult. Even full menu extras aren't difficult; dozens of people have written them. They're just not supported, and apparently for no good reason.

    As for exporting file formats as XML to keep them private: most modern apps that use XML use it because the system does all the heavy lifting for them. Cocoa makes it as easy to save a file as XML as it does in a binary format, just about (yeah, there a few more calls and stuff, but it's not like you have to write expat), and in most cases, there's no reason not to do it for the reasons you mention. Most XML elements aren't position dependent, agreed, and that's a plus. But that doesn't mean that Apple promises that file will always be there with the same elements and attributes that mean the same things. I think Buzz makes this point as well, but he wrote his while I was writing this (I'm seeing it on preview).

    Again, Erik, if you can explain how iTunes changing to an XML format like that described in the PHP article would not break your code, I'd love to hear it. Without that, your argument boils down to "Apple wouldn't change the meanings of the XML elements and attributes in those files." I say that's bilge because undocumented structures have changed repeatedly over the history of the Mac. You're free not to learn from that, but the most successful programmers have.

    And that, whether you like it or not, is the big picture.

    (Since comments are apparently being deleted if they disagree with Erik, I'm not sure I'll be participating here further, but it's been a reascheambake -- arguing about ideas is always cool.)

11. | Reply Steven Canfield
    Posted 08 Apr 2003 at 10:15pm #
    

    I haven't noticed comments being deleted. Erik seems like one of those guys who is for freedom of speech and such, so I'd guess if the post wasn't inflammatory he didn't delete it.

    Matt:

    If Apple documented the iTunes XML file, they would still be free to update the format at any time. You would have to go to Apple's site and look up the changes, and fix your application. I fail to see how you can argue on one hand that the format should be "more" open, but on the other point out drawbacks of that exact idea (that it will break everyone when Apple updates!). The only way to avoid breakage is to totally close (and encrypt) the file format.

    Explain to me, Matt, why it is expected for a shareware developer to update his application for a new iTunes file format but you could not expect such commitment from a commercial software developer? Wouldn't you expect someone like Adobe to be able to put together a patch fairly quickly? Nevermind that being a big developer grants them seeds long in advance, by the way.

    Besides, I don't know if you've heard yet, but Shareware is Dead.

    Lastly, where you note that you have to translate from AW 6 to AW 5 and then translate. How is it, then, that MacLink is able to do that translation on its own (6 => 5)?

    Speaking of MacLink, that is a company that spends a good deal of its time on updates to file formats.

12. | Reply Buzz Andersen
    Posted 08 Apr 2003 at 10:23pm #
    

    All I know is that Michael Tsai had a post, and it is gone now. It linked to this post on his own weblog:

    http://www.mjtsai.com/blog/2003/01/10/cringely_on_macworld_expo.html

    It was posted right after mine, and then disappeared. Maybe it was at Michael's behest that it was deleted, I don't know. I only belabor the point lest people think I am lying.

13. | Reply Buzz Andersen
    Posted 08 Apr 2003 at 10:41pm #
    

    Regarding Matt's last comment: I think it all comes down to Apple's intentions toward third party developers. Is Apple concerned with making things like the iTunes DB available to people like you and me? If so, then they would be likely to make changes to the file format only cautiously, so as not to break anything. And they would be likely to document it (as they have done with Keynote).

    This is why, I think, documentation should be viewed as important in this discussion: I'm sure we're all capable of looking at an XML document and figuring out how it works (although I think this is certainly more true of something like the iTunes DB than a Keynote file), but documentation is still important because it implies standardization, permanence, and (most importantly) a commitment on Apple's part to their third part developers.

    In my opinion, if Apple was really 100% committed to allowing third party access to things like iTunes, we wouldn't be having this discussion right now (and we certainly wouldn't have so many AppleScript-based iTunes add-ons out there). I'm not saying that Apple doesn't have a right to protect their own interest--just that people like Matt and Michael may have some points about Apple's being a (partially) closed shop.

14. | Reply Buzz Andersen
    Posted 08 Apr 2003 at 10:53pm #
    

    Whoops--sorry again (my record isn't very good tonight!). I meant regarding *Steve's* last comment...

15. | Reply Steven Canfield
    Posted 08 Apr 2003 at 11:36pm #
    

    I predict that at WWDC we will get APIs to interface with iPhoto, iTunes, iMovie.

16. | Reply petrilli
    Posted 09 Apr 2003 at 8:55am #
    

    Am I the only one who wonders why people are parsing the XML export to find a file, when the easier way is to send an AppleEvent to iTunes and ask? The XML file is great for some bulk-analysis (like putting out an HTML report of what I'm listening to lately), but it really isn't the right medium for answering questions.

    One of the biggest mistakes I see people making with XML is the same one I saw people making with the whole EDI movement. There's some confusion that XML = obvious. First, if I recall, the format Apple uses is the standard property list format, which is why it's really quite ugly by XML standards---it's designed to be a uniform format regardless of what you're dumping into the file, rather than something more logical by XML standards.

    Having said that, I really really think that people should be using the AppleEvents interface in many cases, rather than the XML dump. If you play with things like the SonyEricsson Clicker, it uses AE, not XML to talk to iTunes to get a list of playlists, songs by an artist, etc., and does so very quickly, without the hassle of having to parse some file that may or may not change. Generally the AE interface (for Applescript and other things in the Open Scripting Architecture) doesn't change much from release to release, if at all. In fact, there's a lot of cruft in some systems to make sure that you don't break anything in the past.

17. | Reply Buzz Andersen
    Posted 09 Apr 2003 at 12:20pm #
    

    petrilli: I suppose you're right, but I guess I've always thought of getting info out of iTunes via AppleEvents/AppleScript as the Mac equivalent of OLE (or whatever the hell they call it now) automation on Windows, which I've always thought was a questionable way to do things (particularly when it comes to creating a robust "commercial" app as opposed to a snazzy hack like the Sony Ericsson clicker).

    It's interesting that you mention that because I'm actually sort of debating how to add iTunes integration to my own app right now. I've considered the AppleEvents approach, but haven't been wild about the idea for the reasons mentioned above. Maybe you're right, though (and maybe it's just an unfair bias on my part).

    On the other hand, if Steven is right (and I hope he is 🙂 the question may be moot after WWDC.

18. | Reply Erik J. Barzeski
    Posted 09 Apr 2003 at 2:46pm #
    

    Some clarifying remarks…

    Much of my argument boils down to the fact that Matt used shitty language. He said "no non-Apple app can" do something that I've illustrated, for a fact, they can. Whether or not a developer may want to is beyond the conversation: Matt says they can't, and I've shown that they can. This makes him wrong.

    Matt uses vague language about menu extras (similar, but not the same as status bars/items). It reads like FUD. Instead of illustrating the reasons (system stability free of third-party problems) why Apple might reserve menu extras for itself, he spreads FUD and his wording borders on flat out lies. iChat, or the messaging framework, as far as Apple is concerned, is a system-level framework. iSync is a core system component. They have menu extras just like the clock or the AirPort thing. Third-party developers have status bars/items. I use them, others use them. It's a good solution.

    Regarding the deletion of posts, as I mention here, I recently switched servers, got a new IP, etc. I think only Michael's comment was lost in the switch - a DNS cache may have been directing him at my old IP address, and thus, his post may have been made to the old server. I've asked him for the content of the post, and will re-post it if I get it, but according to Michael, his point was very small (read what he said on his blog).

    I've deleted all of about four posts, ever, and they were of the "you are a cocksucking motherfucking war nazi asshole" variety. As my "About Me" says, sometimes I bring up points simply for discussion. I don't always take the side with which I agree - I just want to foster discussion and thought, and some good debate. So no, I don't delete posts that disagree with me, because oftentimes I've simply argued a point that I don't even agree with just to get a discussion going (even if it's just a discussion I have with myself).

19. | Reply Michael Tsai
    Posted 09 Apr 2003 at 2:56pm #
    

    Thanks for clearing that up, Erik. I don't think there's any point in re-posting my comment. It was very short, and others have already re-stated what I said in greater detail.

20. | Reply Matt Deatherage
    Posted 09 Apr 2003 at 11:49pm #
    

    I would appreciate, however, if you wouldn't accuse me of living in "what-the-fuck-ville" if you're "just want to foster discussion and thought." My neighbors would object.

    My language was perfectly fine unless you are nit-picking to try to create controversy where none exists, and your increasingly shrill tone reeks of pounding the table instead of pounding the points. I'm not one to be intimidated by boldface, so let's review.

    You can cast 0L as a pointer to a subroutine and call it. You can force closed files that are open by other processes and delete them. You can base your code on undocumented and non-guaranteed data structures. In the end, all three will come to the same end.

    If my word count in Macworld were 3X what it is, maybe I could spend time explaining that "can" means "able to do so in a supported manner that works across a wide variety of computers" instead of "technically doesn't violate the laws of physics." You can open the cargo bay doors on the space shuttle and take a spacewalk without a suit. You won't enjoy it for long, but you can do it. I never considered before now that a sentence like "You can't take a space walk without a space suit" would meet with such heated denial because it's physically possible. Same thing here. "Whether or not an astronaut wants to survive for 30 seconds is beyond the conversation. Matt says they can't, and I've shown that they can. This makes him wrong."

    (And good luck with that semantics degree, btw.)

    The "vague language about menu extras" is not mine, it's Apple's, as linked. It is Apple Computer that tells developers not to make menu item status bars -- including NSStatusBar/Item, btw -- and then does exactly that in its own applications. Whether major parts of iSync or iChat are stored as private frameworks or not is incredibly not the point, though your admirable straw man attempt is noted. Apple does not say "menu extras are unstable," Apple says that developers should not make them because they're reserved for Apple in the human interface. Apple never claims them to be unstable or suitable only for system use, and Mac OS X 10.1 proved that they work quite well; Mac OS X 10.2 revised SystemUIServer to explicitly check the filenames of menu extras and refuse them if they're not on an "approved" list. If you want to assert that anything I've said is a "flat-out lie," Erik, and not just your apologist interpretation, you'd better be prepared to point it out. And according to Apple's guidelines, as quoted (did you bother to read them?), "third-party developers" should not "have status bars/items." Apple says not to do it, not "only do it if it's stable or you like Cocoa."

    You've tried three times now to shoot down the points in the original column, and now you're down to semantic nitpicking on what "can" can mean, ignoring the human interface guidelines, and shouting that this must be right. I think the marketplace of ideas you intend to foster deserves better. And in that vein...

    Petrill: I agree that the Apple event interface to iTunes is probably more stable than the XML file because Apple publishes lots of AppleScripts and highly touts iTunes's scripting ability. However, it's also notoriously slow on large libraries. I'm fairly sure Mac OS X kills the long-time limit of 64K of data per Apple event, but it's still a slow operation to put 4MB or 5MB of data into an AEDesc and pass it back through shared memory. Iterating through a few thousand songs one Apple event at a time isn't any better. Try writing up a C version of your favorite iTunes AppleScript sometime and see if you find it significantly faster on large numbers of songs.

    Buzz: if Apple's willing to come out and say "these are the keys in an iTunes XML library and this is what they mean and they'll stay that way for this version number," that's fine, and is about as much as you can expect from any file format documentation. Then anyone who's already parsing the file can see if he's making bad assumptions, and other programs can add the same features without fear that "improvements" to iTunes will break their code. iTunes is a Carbon application and not implemented as frameworks, so an API to version 3 would be difficult -- but at least all programs could have supportable access to the iTunes library. I maintain that would be a good thing and that Apple shouldn't fear it.

    (I've used bold in several places but it doesn't show up in the preview, just in case the emphasis on "can" doesn't show up in my third paragraph.)

21. | Reply Michael Alderete
    Posted 10 Apr 2003 at 12:34am #
    

    While I don't agree entirely with Matt, I think some of the arguments here, including some of Erik's, are missing the point. While Apple has made an implied commitment to open formats for the iApps, in that there are relatively easy ways to access the information today, Apple has most definitely not made an explicit commitment to the specific formats.

    For some developers, like Erik or the author of Lightbox, this doesn't matter. They're willing to take the (to them) small risk that things will break in the future. After all, their customers are almost by definition online, and actively seeking shareware and other software. They will probably update on their own, especially if they use the software regularly, and it breaks with a new version of Mac OS X. And, if I can be blunt, their paying customers number in the hundreds or few thousands.

    Then there are products like Toast (also mentioned here), with many thousands, probably a few hundred thousand customers, most of whom probably got the product from a physical distribution point (mail order, store, etc.), because Roxio has great distribution for their product. The support cost of the OS breaking Toast functionality is very, very different than for a widget like Bookie. If you don't understand that, you shouldn't be participating in this debate.

    These products are also more complicated, and fixing them when OS changes break features is more complicated and expensive than the case of a single-developer app. Again, if you don't understand this, you probably lack the experience to contribute here.

    I worked as a product manager for an operating system company (Be Incorporated). We had direct, regular experiences where developers used immature or unsupported formats or APIs in their applications. When we changed them it was a Big Deal to those developers, sometimes to the point of crippling them. No shit. If a small developer strikes out on their own, and spends 6 months building a great app, that gets broken by an OS update, they may not have the savings left to be able to recover. You can kill a developer like that.

    Matt has clearly taken a controversial, discussion-provoking position -- as any good opinion columnist will do on a regular basis. Some of his comments are not entirely correct, but his overall argument is certainly not entirely wrong. Matt explained that he worked in Apple's DTS organization, and saw the consequences of developers depending on file formats and APIs which were not supported, and then changed, many times over. Don't dismiss his experience as "FUD" or "bordering on lies"; that's being intellectually dishonest.

22. | Reply Erik J. Barzeski
    Posted 10 Apr 2003 at 12:52am #
    

    From the Apple document you seem happy to quote so quickly: "Last Updated June 2002". Gee, that would be… before Jaguar shipped, and before Apple created a supported method - NSStatusBar/Item - to do menu bar items.

    You're wrong about the XML DTDs and whatnot. You're wrong in assuming that'd "break" an app where any developer worth their salt would put in a check on a nil key or something. Have developers done this? No. Can they? You bet your ass they can. OmniGroup is. I have. Josh has. Others have too.

    Part of my argument may be a semantic one, but your "space suit" argument really doesn't help your cause at all. It's a laughable analogy. And as for the insults, well, I'll continue to make mine blatant in favor of your backhanded "good luck with your semantic degree" variety.

    Anyway, do yourself a favor and look at iPhoto's XML file. See the versioning? See the major/minor versions? I'll give you one thing: it's a bugger that this isn't "official" in some capacity, but guess what? Even if Apple published a spec there'd be nothing to stop them from upgrading the format or improving it in the future. It's published in a DTD'd XML format. That's as open as it needs to be to allow developers to work with it. Third-party developers have and will continue to do so.

    You remain wrong, Matt. But hey, feel free to continue to delude yourself. 🙂 After all, it's a free country.

23. | Reply Steven Canfield
    Posted 10 Apr 2003 at 1:27am #
    

    Not to rob the grave here, but I have a question. NSStatusBars have been done, to death. People have proven they can access the iTunes XML file. People have proven they can access the iPhoto XML file. People have shown that this XML stuff _IS_ self documenting, because every tag has a name. Every one. Just in the way if you looked at a HTML page without knowing HTML, you can understand that the junk between the tags is going to be the page title, you can read XML. If you can't, you're daft. Or, the programmer has gone out of his way to name his keys something odd, JUST to screw you up. Look at the iTunes XML: things like "Name", "Bit Rate", and "Artist".

    And so the point is this: Apple may not be officially supporting these things, but it's perfectly within your grasp to use them. If Toast wanted to read your playlist, they could. That they don't is either a knock against them for laziness or proof that they don't want to.

    Matt, your comparisons make no sense. How often, in the development of a piece of software (perhaps a less well planned one) does the file format change? I'd say fairly often. Developers can cope.

24. | Reply Matt Deatherage
    Posted 10 Apr 2003 at 2:51am #
    

    Once upon a time, back in the olden days, a developer wrote in to our group complaining that his application didn't work properly. The toolbox was malfunctioning, and could we please figure out what the problem was.

    Back in those days, toolbox dependencies were a bit tricky, and the tools had to be started in a certain order, not like today where the OS initializes everything for you (more RAM is a wonderful thing). The required order was carefully documented, but these developers had almost completely ignored it. By starting some tools before others they required were available, the entire system got confused.

    One of our engineers spent some time on behalf of the developer and rewrote his toolbox initialization code, with all the calls and allocations in the right order. He plugged it into the developer's program, and like magic, every problem the developer had complained about went away. Every bug they had blamed on the toolbox just vanished when everything was done properly. So the engineer sent the code back to the developer with the assurance that this would fix every problem they'd reported.

    The developer responded the next day: "Thanks, but we like the order we were starting the tools in. Please tell us how to fix the problems without changing the tool initialization order."

    The moral of the story: if a programmer is bound and determined to believe something, no amount of logic will convince him otherwise. If you folks want to continue to believe that basing a product's main features on unsupported data structures is just fine because you are sure you understand every possible value and what it means, that's your choice.

    It's also a red flag warning to anyone who's been involved in professional computing not to take your software seriously. How do you know to check for a "nil key" if you don't know if NIL is a legal value? How can you check against a DTD that doesn't exist? I see the versioning in AlbumData.xml -- what does it mean? Do you know that a change in the minor version means all the elements remain the same? What about a change in the application version? In the build number?

    No, you don't know. You're guessing, and you're betting your users' productivity that it won't break. If you want to laugh at that, be my guest. You should just hope potential employers don't read you advocating it before any job interviews. I'm obviously not going to convince you that using undocumented and unsupported data structures is a bad idea, so have all the fun you want doing it. Just have the courtesy to tell your users what you're doing so they don't find out the hard way with iPhoto 3 or iTunes 4.

    Steven: no one's saying that file formats don't change during development. They even change between versions. In fact, that's the entire point.

    (By the way, although the comment system says the "b" tag is supported, it doesn't do anything; you have to type the much longer "strong" to get boldface. Gotta love the Cocoa way.)

25. | Reply Erik J. Barzeski
    Posted 10 Apr 2003 at 10:50am #
    

    Matt, your continued inability to simply read what's written persists in amazing me. All of a sudden we're making the integration with iTunes the "main feature" of a program? Do yourself a favor and figure out where anyone said that. It's hardly the main feature in iMovie, let alone a third party app. It sure as hell ain't a main feature in Lightbox.

    Potential employers? Get off your high horse man. You're wrong, you've been proven to be wrong, and you're going onward anyway. Get over yourself.

    You've ignored nearly every point put forward, instead going back to assertions that your "new" definition of the word "can" is fine, that developers haven't done as I've suggested, and all sorts of things. You're relying on stories about astronauts and mythological stupid developers who can't write toolbox code. Hearsay won't win you points. You're still sitting squarely in the negative.

    P.S. WTF does Cocoa have with your inability to realize that the <b> tag works fine. Nothing. Zip, zilch. It works fine, by the way.

    P.P.S. Neener neener, now I have the last word. 😛

26. | Reply Deatherage a Doozy? | NSLog();
    Posted 10 Apr 2003 at 11:42am #
    

    It seems Matt Deatherage's May1 article, the one following his error-laden April article, has also caused some debate, this time with Watson developer Dan Wood....

27. | Reply Anonymous
    Posted 18 May 2003 at 4:08am #
    

    You're new to the Mac, right? Mac developers expect APIs they can rely on. Documentation so their applications won't break with the next update of the OS. NeXT-developers seem to be happy if they can just find some way to do something.

    Using XML doesn't make your file format open or readable. XML is a meta language. A file format based on XML isn't a priori any more open or readable than any other file format.

28. | Reply Erik J. Barzeski
    Posted 18 May 2003 at 9:28am #
    

    I've been using Apple computers since before the Mac. So no, you couldn't be more off.

    BTW, thanks for having the balls to leave a name and an email address or Web site.

29. | Reply Anton Rang
    Posted 19 May 2003 at 11:39pm #
    

    Steven wrote:

    People have shown that this XML stuff _IS_ self documenting, because every tag has a name. Every one. Just in the way if you looked at a HTML page without knowing HTML, you can understand that the junk between the tags is going to be the page title, you can read XML.

    Hmmm… somehow it's not intuitively obvious what this extract from my iTunes Music Library indicates:

    <key>Smart Info</key>

    <data>

    AQEAAwAAAAIAAAAZAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

    AAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAAA

    AAAAAA==

    </data>

    On the other hand, I tend to be slightly less enamoured of "open formats" than Matt's position in his column. I think there's a solution which is the best of both worlds: A documented file format, and a shared library to access it.

    Documenting the file format means that it can be used on another platform, or after the original software has long since passed into oblivion. This is (almost self-evidently) a good thing.

    What's not so obvious is that documenting the file format can also lead to stagnation. If Apple documents exactly what "Smart Criteria" is, and third party developers begin to rely on that, then Apple can't change this key because it would break third-party applications.

    This is where having a shared library (a framework, for Mac OS X) comes into play. Third parties can use a well-defined API to access files; if Apple updates the file format, they can simply update the API as well. Applications may require updates to take advantage of new features, but they won't break if the file format changes. (The keychain is an example of how this can work well in practice.)

    Providing a shared library allows applications to access files in a supported way; providing file format documentation allows developers on other platforms (or other OS versions) to read and write those files as well.

    Building and testing a shared library is much more work, but it's likely that Apple's doing much of this anyway with iLife; hopefully they're not actually writing separate code for iPhoto, iDVD, etc. to access the iTunes library, for instance. Much of the additional work involved would be the documentation (which is, however, a lot of work).

    Using reverse-engineered file formats is sometimes necessary, but it's always risky. Apple should not be encouraging developers down this path; it's bad for the developer, bad for Apple, and bad for the platform.

30. | Reply Anton Rang
    Posted 20 May 2003 at 1:01am #
    

    A few more examples of how XML is not self-documenting, with examples drawn from iTunes....

    Without supporting documents, the range of values isn't known. For instance, the track ID is listed as an integer. Are negative IDs allowed? Is zero reserved? What about 1? What are the legal values for Genre (a "string")?

    The domains from which values are chosen, and their semantics, aren't known. For instance, if playlists had IDs (and it's odd that they don't), could a playlist ID be the same as a track ID? I can think of plausible implementations which would prohibit this, as well as ones to allow it. For that matter, can a track number be greater than a track count in the same entry? (I have one CD with 12 "listed" tracks and a 13th "hidden" track....)

    The semantics of XML elements aren't known, either. Is it legal to have a playlist with no tracks? What about a playlist in which a track appears more than once? What does it mean when a playlist has "All Items" set to false?

    XML is just a way of writing an association list for people who can't follow LISP. 😉

    (Also worth noting, though a separate point, is that the iTunes XML representation is still missing a number of fields from the real library, such as equalizer settings.)

31. | Reply Michael Tsai's Weblog
    Posted 17 Jan 2004 at 12:59pm #
    

    Games

    Halo didn’t initially work on my iBook G4; it would always quit just after showing the splash screen. After some weeks, Greg Grimes at MacSoft figured out that I should copy the “shaders” folder on the CD into the “Halo” ...