[This post is one in a series of posts on Sente 6, the next major update to this software. Sente 6 is scheduled to be released in early Fall 2009. Everyone who purchases Sente 5 after January 1, 2009 will receive Sente 6 as a free upgrade.]
Over the past few months, we have released a couple of private previews of Sente 6 and received valuable and encouraging feedback from our testers. There will be a few more private previews before our first public preview, but we are getting close enough that I wanted to begin saying a bit more about the design of Sente 6.
In this post, I would like to discuss an important change to the way in which Sente 6 can deal with attachments. (Attachments are any files that you link to a reference in Sente. These are most commonly PDF files, but they can be any file whatsoever.)
First, let me say what is not changing. As in previous versions, in Sente 6:
- Any number of attachments can be linked to each reference.
- Any types of files (not just PDFs) can be attached to references.
- Attachments can be automatically renamed and filed according to user-defined rules, or simply linked to.
- When attaching a web page to a reference, Sente can automatically create a web archive of the page and file the archive appropriately.
A new option in Sente 6 is to have Sente copy (or move) new attachments into the library itself, rather than leaving them in the outside filesystem. (Libraries in Sente 6 are actually OS X "bundles", which are really folder hierarchies that appear in Finder to be a single file. When attachments are placed inside a Sente 6 library, they are actually stored as files within the library bundle.)
There are several reasons why people might want to use this new option.
One, this makes it dramatically easier to move or copy a Sente library from one computer to another. With Sente 5 (and earlier), moving a library from one machine to another meant copying the main library file (from ~/Library/Sente) and all of the folders containing attachments (typically in ~/Documents). And if anything about the file paths changed along the way (e.g., the user's home folder changed names) attachments could break. Users have had to spend far too much time moving their data from one machine to another over the years; storing attachments in the library bundle makes this easy.
Two, this makes operations like backing-up a library, or copying a library to an external device (e.g., USB drive) very simple. Do you want to give a copy of your library to a colleague? Just drag the library file onto a thumb drive and everything, including PDFs, is there.
But the most important reason for bundling attachments is that it supports the library replication mechanism in Sente 6. I will be writing more about replication soon, but for now I will just say that I think all of you who have struggled with maintaining one library across multiple machines will really like the way replication works in Sente 6. It will change the way you use Sente.
Now there are also some reasons why people will be hesitant about storing attachments in Sente library bundles. For example:
Many people will worry about being able to get their PDFs out of Sente should they decide to switch to another reference manager in the future. To address this we are adding the ability to easily export some or all attachments at any time. We will place them wherever you want them, and name them according to your wishes.
Some users want to have multiple programs access their attachments. For example, many Sente users also use DEVONthink on their PDFs. We are still working on exactly how this interaction will be facilitated, but we suspect that some sort of direct access to attachments within a bundle will be needed.
There will also be many users who simply like being able to browse their PDFs in a nice folder hierarchy of their own design and they would feel lost if this were to disappear from their Documents folder. For these users, it might be sufficient to store a copy of each PDF in Sente; disk space is cheap and one can store more than 1,000 typical PDFs per gigabyte of disk space, so this is not that much of an issue. The problem is worse if you routinely mark-up your PDFs (e.g., in Acrobat, Preview or Skim); then you may not want two copies. But even in this case, we think there are approaches that will work.
In the end, of course, the choice of whether attachments are stored in Sente 6 bundles or not will be up to the user. If you like the way thing work now, you can continue using the older approaches. You will still even be able to take advantage of Sente's new replication system. But if you want your PDFs (and not just your references) to magically appear on each of your computers, you will need to store them in the Sente library bundle.
Let us know what you think about storing attachments in bundles. Do you have other issues that you think we need to consider? Suggestions for how Sente should work with other software in managing attachments?
Stay tuned: I will follow up soon with additional postings about Sente 6.
Michael
Michael,
In my view, the crucial question it whether storing PDFs in Sente bundles will make them invisible for Spotlight or not. I use Spotlight all the time to look for things I know I have read somewhere but cannot remember exactly in which article or book.
Plus, the PDFs stored in bundles would only be accessible from within Sente, right? To open a PDF I would always have to open Sente, load the library, launch a query, select the pdf and open it (no shortcuts). Unless Sente 6 is much, much faster than Sente 5 this would be a burden for me. I am used to opening my PDFs with QuickSilver in a fraction of a second.
I am also a little skeptical about your suggestion to use different copies of the same PDF for use inside and outside Sente. How would you make sure that the annotations, highlights, etc keep in sync?
Yes, I failed to mention that you will have the option of making all attachments available to Spotlight, even though they may be stored in the bundle. In this way, it will be like Mail, where you almost certainly only access messages through the Mail program, but because they show up in Spotlight, you can find them from outside the program.
And the approach of keeping two copies if not best if you regularly mark-up your PDFs because there is no good way to keep them in sync.
Michael
Michael,
How will this mechanism handle large numbers of references with attachments?
I currently have 6100 references with PDFs attached.
Thanks,
John
Hi John
We have not yet tested with very large collections, but there is not reason to believe that performance will be any different than current performance. Bundles are just folders to everything but Finder, so there is really no different from that perspective. Other changes in Sente 6 should help with performance overall, but I do not see any reason why bundling attachments would make much difference one way or the other.
Michael
Will the pdf files stored in the bundle be renamed and organized according to the user preferences, or they will have a weird organization/naming scheme, like the early versions of iPhoto?
Michael, you write that spotlight integration will be available. Why don't you bring it inside Sente? I gave up using spotlight to search my PDF library because using the Finder to go through the results is a bit messy (well, I still use it if I have no other choice...). And I am sure that sente would handle the spotlighted PDFs in a much better way.
BTW all the major competitors (Bookends and Papers) have this feature since a long time...
Silvo
PS if you decide to bring spotlight in Sente, please keep reference search and PDF search separated: ie if I search for an author, I don't want in the results all the papers that cite his work...
Smart moves -- bundles and replication -- and equally smart to preserve user choice in case the new system doesn't mesh with a user's workflow. Just a few thoughts:
1. In addition to the attachment export functionality you are developing, it is worth remembering that one can open a bundle as a folder (i.e., in the Finder) by right-clicking on the icon and choosing "Show Package Contents." It might be useful to make sure that the Attachments folder is a top-level folder within the Sente library bundle, so that a user could easily navigate to (and within) it that way. That might alleviate some fear of adopting the new structure. Also, it should be possible to place an alias to that Attachments folder on the desktop, or in the dock, for quick Finder-level access to bundled attachments. Of course, it would presumably be important that users not add/delete files directly to/from that folder but rather let Sente handle it.
2. Over in the forums, there was a recent thread (http://www.thirdstreetsoftware.com/forums/showflat.php?Cat=0&Number=5713&an=0&page=1#5713) about the potential problems associated with the storing of custom fields and status labels in the preferences file. The take-home message was that is is important to back up the prefs file to avoid loosing custom fields, and to remember to copy this file over when transferring a library to another computer. Hopefully, the Sente 6 bundle-approach will allow a better solution to this. A significant risk with the present system is that preference files can get corrupted. ( I've developed a habit, whenever I apply an update [e.g., from version 5.x.1 to 5.x.2] of backing up the Sente.plist file first, lest I need to go back to the previous version for some reason and find that the new pref file is not backward compatible, lest I loose my status labels and custom fields in the process.)
3. I agree with Silvo's suggestion about Spotlight search within Sente (and about being able to either limit searches to just reference metadata or include attachment content, to fit the specific search need!). Do I remember, however, reading somewhere that Sente 4 had internal Spotlight searching and that it was eliminated in Sente 5 due to performance issues? (I upgraded directly from version 3 to 5, so never experienced version 4.)
Looking forward eagerly to Sente 6...
-Mark
I have a few questions regarding how the bundles will affect various workflows:
How will the bundles affect workflows associated with emailing PDFs as attachments?
How will we be able to locate the PDF from within our email application so that we can attach it to a message?
Will there be any differences in behavior that are dependent upon the email app? For example, I can envision a scenario in which Sente could tell Mail.app to open a new message with the selected PDF attached, but this might not work for gmail users (like myself).
Finally, on a different topic, it looks like when Snow Leopard comes out, Apple's Preview app will allow sensible selection of text in multicolumn documents (http://www.apple.com/macosx/refinements/). Will this be available in Sente?
Thanks,
John
I have another question....
What about backing up the new libraries with bundled PDFs?
Will Time Machine try to back up 1Gb of almost identical data every hour?
Thanks,
Silvo
Hi Silvo,
The short answer as to how PDFs will be named is that they will not have names that are meaningful to users. We were hoping that we would be able to give them meaningful names, but replication made this impossible. We are planning on making it possible to have links placed elsewhere with meaningful names, so users do not ever have to deal with the random-seeming names we will be using within the bundle.
And, yes, we will be supporting searching within PDFs within Sente.
And, finally, Time Machine should be fine with the PDFs in the bundle. If they were placed in the SQL database (which is a single, large file) this would be a problem, but as individual files, TM should handle it just fine.
Michael
Hi John
Emailing references and attachments is much improved in Sente 6, at least for Mail users. Gmail presents some problems for us, but we are still working on this.
And, we hope that Apple's PDFkit will make selection in multi-column docs easier, but we do not yet know the answer.
Michael
Hi Michael,
I don't want to discourage you from continuing to work on the problems with emailing references for users of conventional, browser-based Gmail, but would it be easier to work out using the stand-alone Gmail app, Mailplane (http://mailplaneapp.com/)?
I demoed Mailplane for a month and quite liked it (especially the fact that it let me drag-and-drop attachments and full integration with Apple's Address Book), but this wasn't quite enough to get me over the hump of paying for it (though its not really expensive). If I can get the benefits of Sente 6 while using it, that would be enough. Maybe you could do a package deal with them....
John
Hi Michael,
I don't want to discourage you from continuing to work on the problems associated with emailing refs and attachments with conventional, browser-based Gmail, but would things be easier with the stand-alone Gmail app, Mailplane (http://mailplaneapp.com)?
A while back, I tried the Mailplane demo and I quite liked it -- especially the drag-and-drop attachments, the handling of mailto links, and the integration with Address Book -- but these weren't quite enough to induce me to buy it (though it's not expensive). If I could use it to get the benefits of Sente 6 while still using Gmail, I'd buy it.
Maybe you could partner with Mailplane on a bundle.
John
Sorry for the double posting -- it looked to me like the first had been lost....
John
Hi Michael,
I don't know if I've understood correctly, but do you mean that attached pdfs will not have the names you create with the present filing/renaming system? Does that mean that if I index them with DEVONthink the file names will be "meaningless"?
I hope I've misunderstood you.
Thanks
Manuel
Hi Michael,
I think the effort of working on library sync and sharing is well invested, it is indeed a priority.
I also appreciate the tradeoffs this might entail. Leaving options open for users is, therefore, important. In my current use, I have found a workable solution for syncing across two computers. It is rather the process of filing attachments that I have found a bit cumbersome. I find myself constantly switching "let Sente manage attachements" on and back off in the preferences.
This is because I like Sente's ability to create new references, download the corresponding PDF and attaching it on the fly. I have created a special reference type to rename the attachment with a simple and clear name (mainly APA5 but without punctuation).
And then I switch managing files OFF in the preferences, because I don't like any of the folder structures that Sente proposes for automatic filing. So I just move the new, rightly named attachment to the folder I want in my own thematic folder structure (the link in Sentes is preserved when moving the file). I want to be able to browse and edit and comment attachments outside of Sente. And I DONT want Sente to refile all my PDFs, two thirds of which aren't linked in Sente anyway. Plus, when I drop an (existing) PDF onto an (existing) reference in Sente, I dont want Sente to duplicate the already existing reference in the default location, and then link the duplicate. I just want it to link the existing reference right where it is. A lot of this stems from the fact that I manage a legacy of references and documents which weren't originally created in Sente. I might not be the only one in that case.
As a result, any solution that would make Sente behave more monolithically, becoming the single interface for opening attachments and giving them obscure names, is just a no-go for me, because I also manually save lots of documents in my documents folder, which aren't necessarily linked. I would rather wish a more flexible approach, where Sente would download and rename new attached documents, but wouldn't meddle with documents already on the HD, but would just link them. This would stop the hassle of turning "Manage linked files" ON/OFF.
But maybe I'm just a single case in you users? I'll post this on the forum to see reactions.
Kind regards,
Jan
Would it also be possible to ask sente to locate the different library on different disks ?
Some of us are not working on recent mac or do not always have enough disk space to carry all their libraries on their laptop especialy when they are very big.
Yes, you can place each library on a different disk, if you want to. Also, in the case of synchronized libraries, you can set up the copy on your laptop to download attachments on demand, rather than whenever they are added to the library. In this case you will see that there is an attachment and clicking on a link will bring it quickly to your laptop for reading. You can even then delete it from the laptop and repeat the whole process again if need be.
Michael