The Sente Blog

The bottom line is that we are excited about the new platform and we will release a version of Sente for the iPad as soon as we can.

We are not yet prepared to commit to any particular functionality, but we think some of the key characteristics of iPad Sente are rather obvious, so there is little advantage to keeping them secret.  Here are some of the key features that we expect the application to include:

Mobile libraries synchronized with desktop Sente.  Frankly, many of the design decisions that went into synchronized libraries in Sente 6.0 were made with the rumored tablet in mind and we think Sente's synchronization mechanism is well-suited to the task.  One library, multiple devices.  New references and edits show up everywhere almost immediately.  And "download on demand" for attachments means that you can have as many or as few PDFs on your iPad as you want, and quickly obtain any others as needed with a single click.

On-the-go PDF reading, markup, tagging and note-taking.  The iPad is clearly much better suited to reading PDFs than the iPhone.  To make this most useful, it is essential that users be able to do markup, note taking and tagging as they read.  And, of course, all markup, notes and tags will be quickly and automatically synchronized.

We would not normally make such public statements this early in the development cycle -- one never knows exactly what kinds of unexpected hurdles will be found as we work on the software --  but in this case we think our users should know where we are headed.  We have been hoping for and planning for the iPad for some time, and we are very excited to be able to begin serious development.  

Anticipating the next question... we do not yet know when this application will be released.  When we know more, we will tell everyone, but right now we simply do not have a schedule.

Now we want to hear from you.  Would you like to see Sente on the iPad?  How would you use it?  What do you think the must-have features are?

Michael

First, some background. In 6.0, if you asked Sente to store attachments within the library bundle (which is required if you want attachments synced) they were given a name based on an internal identifier.  This was great for us because it let us sync attachments with confidence that we were not going to accidentally overwrite files on any of the machines on which a synchronized library was installed.  However, it caused a number of problems for users, including:

This is changing in Sente 6.1, due out in a few weeks.  In this new release, you will be able to specify how attachments will be filed and named, much like in Sente 5.  For example, you can have your files organized by journal, year, author and then named with the title of the paper.  Or organized by year and named with the author's name + year + title.

Also, you will be able to have attachments filed outside the library bundle and still synchronized.  So you can place your PDFs in ~/Documents/Sente instead of the bundle and still have them synchronized across multiple computers.  Further, the location can be different on each synced computer.  It can even be in the bundle on some computers and outside the bundle on others.  The folder hierarchy has to be the same on all computers, but the starting location can be different on each.

There is a catch to this, however.  We have to know that the file 2007/Smith/My Paper.pdf is intended to be the same on all computers because when it is updated on one computer, it will be copied to all others, and we have to be confident that we can simply replace the similarly filed/named file on each machine.  For this reason, the folders in which Sente attachments are filed must not be used for any other files.  That is, you cannot intermingle other PDF files with your Sente PDF files, for example.  For this reason, when filing outside the bundle, Sente will always insert a folder with the library name in the path.  For example, if you specify ~/Document/Sente as the root folder for filing attachments for "My Library" Sente will start the filing in ~/Documents/Sente/My Library.

There are some other limitations on the new approach.  For example, the file naming patterns are hard-coded into Sente (as opposed to being bibliography formats, as in Sente 5).  This is done because we have to know that the naming scheme is supported on all synchronized copies.  We will add new options based on user requests, but you will not be able to create your own formats using bibliography formats.

Sente 6.1 also includes the ability to refile all existing attachments based on a new scheme.  For example, you can convert from having PDFs organized by journal and year, to being organized by author and year.  Sente will automatically move and rename all attachments for you.  However, this cannot be done once synchronization has been turned on for a library.  If you need to do this, you will need to make a new copy of the library, set up attachment filing the way you want it, and then turn sync on for the new library.  You can move the entire attachment hierarchy on any copy of a synced library (e.g., you can move it from within the bundle to somewhere outside the bundle, or from Documents/Sente to Documents/PDFs); you just cannot change the organization or naming below that level once sync is turned on.

Here is a shot of the attachment configuration screen:

One other change we have introduced since 6.0 was released is that we no longer support "mixed state" for attachments.  That is, if you have told Sente to file all attachments inside the bundle with a particular filing/naming scheme, all attachments in that library must be filed in that way.  You cannot have any attachments in that library stored anywhere else.

We are still encouraging users to store attachments within the library bundle because it is much easier to make many common changes (e.g., just move the entire library from one location to another and everything will still work).  But we are also making it easier to handle changes like the migration from one computer to another even when the attachments are filed outside the bundle.

We are currently testing these changes internally and expect to make them available to a few outside testers in the next week or two.  Not long after that, we will make the official release of Sente 6.1 with these changes.  

But I would like to make one request: Please do not extend existing threads with questions that are not directly related to the title of the thread.

I know how easy it is to just hit Reply and say, "while I have your attention..." but this has two very negative effects:

First, it makes for long threads, which take a lot more time to scan through before we feel like we can answer the last message in the thread.  This just slows us down and means that support for other issues suffers.

Second, it means that when we look through the list of outstanding issues, we cannot tell what the subjects are without reading the entire thread.  The original topic might have been "migrating data from Sente 5"  but the new topic is "how tags work."  It is not hard to see how this makes support harder.  

Many users are already in the habit of starting new threads for new topics and we appreciate that.  Some others may read this and pause before writing "oh, and one other thing..." and we appreciate that.

For the others, this post is just something we can point to when we send out a reply to a change of topic with a request that a new thread be started.  It is hard to write that in a way that does not offend anyone, but we are not trying to be rude.  We are really just trying to keep up.

Michael

When you open a PDF in a single reference tab, you can highlight text or a region (just like previous versions) and you have the choice of:

1) creating a note from the selection (like earlier releases), or

2) highlighting the text selection (or boxing a region selection), or

3) doing both at once

This is obviously a limited subset of what Preview or Skim offer for PDF markup, but we suspect that these are among the most commonly used markup commands.

One of the nice aspects of doing the PDF markup in Sente is that, at least for PDF stored in the library bundle, the markup is synchronized across all copies of the library.  So if you have a synchronized library, any markup performed on any copy will automatically appear in all copies shortly thereafter.  (Because of the size of a PDF file, changes are synchronized only when the single-reference tab is closed; this helps limit the amount of data being sent back and forth between computers.)

The markup that Sente adds to a PDF is compatible with Preview, but not Skim.  (Skim uses a non-standard method for storing annotations on a PDF file.)

We think having attachments synchronized across computers is pretty cool, and replicating markup is, well, even cooler. Try out the beta and see what you think.

Michael

Here are some of the highlights of Sente 6.0:

So, what is tagging and how does it differ from keywords?  Well, there is no universal answer to this question, but here is our answer to the question.

1) In Sente 6, keywords are words or phrases attached to references by external data sources (PubMed, Web of Knowledge, JSTOR, etc.).  Tags are words or phrases assigned to references by the Sente user.

2) In Sente 6, tags can either be created and assigned "on-the fly" by simply typing new values into the appropriate field in the reference editor, or assigned from the new QuickTag palette.

3) Tags in the QuickTag palette can be organized hierarchically.  That is, one can organize QuickTags into categories that nest arbitrarily deep.  Note that this is different from many tagging systems that do not permit nesting of tags. 

For many people, it will be most efficient to work with tags using the QuickTag palette window.  This window displays the entire hierarchy of QuickTags and shows which tags have been assigned to the currently selected reference(s).  Toggling tags in this window will assign or remove a tag from the selected reference(s), which makes it easy to assign tags to many references at once.

In Sente 6, tags can be used in Smart Collection definitions and Sente automatically creates a hierarchy of built-in smart collections matching the QuickTag hierarchy, so you can easily see all the references that include any particular tag.

One thing to note about the QuickTag hierarchy is that if a reference is tagged with a tag several levels down in the hierarchy, it behaves as thought it is tagged with all of the parents of that tag.  For example, if the QuickTag list included a category called "Bicycles" with sub-categories of "Mountain Bikes" and "Road Bikes", any reference tagged with "Mountain Bikes" would be included whenever references tagged with "Bicycles" are displayed.  And these "implied" tags are evaluated on-the-fly, so if you were to change the hierarchy, the new structure would be used when determining which references should be included.

So, enough text about tagging.  Here is a brief video introduction to the new feature:

A public preview of Sente 6 will be released in the coming weeks and officially released a few weeks later.  Everyone who purchases Sente 5 now will receive Sente 6 for free when it is released.

One improvement worth noting that comes along as part of the Snow Leopard upgrade is that selection of text within multi-column PDFs is much improved.  If you run Sente under Snow Leopard, you will find that selecting text from one column in a multi-column PDF now works the way you thought should have worked before.  Hats off to Apple on this one.

Michael

Today, we released Sente 5.7.9.  There were a number of small changes in this release, but the primary reason we had to get this out is because Google changed the structure of the HTML in their Scholar pages and these changes broke our reference detector code (the code that puts the little target icons next to each reference).  So, for the last couple of days Sente users have been unable to get references from Google Scholar pages.

Sente 5.7.9 fixes this problem, but the underlying problem is the lack of a stable API to Google Scholar (and to many other resources).

So, what exactly is an API?  It is an interface for use not by people, but by programs.  A user interface, or UI, is designed to be used by a person. An API is designed to be used by other programs.

An example might help.  If you use Apple's Mail program and you have a MobileMe account, you can access your email either through MobileMe's web-based user interface, or you can read your email in Mail.  The Mail application uses several APIs at MobileMe to get and send mail (e.g., SMTP, IMAP, POP).  If Mail could not use these APIs, it would be forced to try to extract mail messages from the web interface, which would be very difficult, slow and error prone.  And, every time someone at MobileMe tweaked the way the web page looked, the Mail application would have to be modified to handle the changes.

This is where we are with respect to Google Scholar and hundreds of other data sources.  Our targeted browsing code looks at the structure of each web page that it gets and tries to find references embedded within the page. This feature can be very useful, but it is not nearly as useful as it could be if each site supported some standard APIs for searching and retrieving reference data.  And, every time one of the supported sites changes their HTML, we have to modify Sente to handle the change.

Contrast this with PubMed.  PubMed has supported a well-defined, relatively stable API for accessing reference data (and other types of data) for many years now.  This means that Sente can do more things, more quickly and more reliably, with PubMed than we can with most other data sources.  And our users in the bio-medical sciences benefit directly from this.

People in the humanities often accuse developers of academic reference managers like Sente of caring more about biology than about French literature, or medieval art.  This is not the case.  The problem is that the data sources in the humanities (and many other fields) are far, far behind the data sources in biology.  

Instead of treating citation data (titles, author names, abstracts, publication details, etc.) as a little advertisement that will attract readers to their publication, many publishers in the humanities (and other fields) treat citation data as an asset that they can license to services like Thomson and EBSCOhost, who in turn sell access to institutions.  I think it is fair to say that the primary motivation for this way of doing things is not to provide the best support possible for academic research.  (Note that I am not talking about free access to the full text of all articles; just to the basic citation data about each reference.)

Now, you might be thinking that I have confused two issues -- APIs and the licensing of reference data -- and, to some extent I have.  But they are related.  Google provides an API for regular Google searches.  But not for Google Scholar.  Why?  Because the academic publishers would not agree to this because they were afraid of losing their income from licensing their data to resellers.  Thus, we are reduced to trying to keep up with each little change in the HTML that Google makes to their Scholar search results page.

So, back to the original question: why should you care about APIs?  Because the lack of support for open, stable APIs to basic reference data in many fields is holding back everyone  those fields.  We will continue to find ways to make it easier to search the literature in all fields of academic study, but we would be more successful if the publishers actually wanted this to work.

Many of you have, or will have, significant influence on one or more of the academic publishers in your fields.  You may be on the editorial board for a journal.  You decide which journals you submit your manuscripts to.  I think it is time for everyone to use whatever influence they have to push publishers in all academic fields to follow the lead of publishers in the bio-medical sciences.  It is in everyone's best interest (even the publishers) that there be ready access, through both user and programming interfaces, to all of the basic citation data in all academic fields.

The following video demonstrates the use of synchronized libraries on one computer.  The mechanism is exactly the same when the copies are on different computers, but this was the best way to show the updates as they happen.

In the real world, performance will be a bit slower than in the video because you will be further from the servers than we are, but updates will still propagate in just a few seconds.  The biggest difference will be seen when first synchronizing a large library and when propagating PDFs -- which are much larger than typical reference updates -- these will take quite a bit longer than in the video.  But other than when you first create a synchronized library, the volume of data going back-and-forth will be relatively modest, so you should still be pleased with the performance.

Synchronized libraries is working in Sente 6, Preview 3, which was recently distributed to our small group of testers.  We will be making one or more public previews available in the coming weeks, so everyone will be able to kick the tires shortly.

Please let us know what you think -- we want to make sure that our design for synchronized libraries works well for as many people as possible.

Michael