Recently in General Reference Management Category

Today, we released Sente 5.7.9.  There were a number of small changes in this release, but the primary reason we had to get this out is because Google changed the structure of the HTML in their Scholar pages and these changes broke our reference detector code (the code that puts the little target icons next to each reference).  So, for the last couple of days Sente users have been unable to get references from Google Scholar pages.

Sente 5.7.9 fixes this problem, but the underlying problem is the lack of a stable API to Google Scholar (and to many other resources).

So, what exactly is an API?  It is an interface for use not by people, but by programs.  A user interface, or UI, is designed to be used by a person. An API is designed to be used by other programs.

An example might help.  If you use Apple's Mail program and you have a MobileMe account, you can access your email either through MobileMe's web-based user interface, or you can read your email in Mail.  The Mail application uses several APIs at MobileMe to get and send mail (e.g., SMTP, IMAP, POP).  If Mail could not use these APIs, it would be forced to try to extract mail messages from the web interface, which would be very difficult, slow and error prone.  And, every time someone at MobileMe tweaked the way the web page looked, the Mail application would have to be modified to handle the changes.

This is where we are with respect to Google Scholar and hundreds of other data sources.  Our targeted browsing code looks at the structure of each web page that it gets and tries to find references embedded within the page. This feature can be very useful, but it is not nearly as useful as it could be if each site supported some standard APIs for searching and retrieving reference data.  And, every time one of the supported sites changes their HTML, we have to modify Sente to handle the change.

Contrast this with PubMed.  PubMed has supported a well-defined, relatively stable API for accessing reference data (and other types of data) for many years now.  This means that Sente can do more things, more quickly and more reliably, with PubMed than we can with most other data sources.  And our users in the bio-medical sciences benefit directly from this.

People in the humanities often accuse developers of academic reference managers like Sente of caring more about biology than about French literature, or medieval art.  This is not the case.  The problem is that the data sources in the humanities (and many other fields) are far, far behind the data sources in biology.  

Instead of treating citation data (titles, author names, abstracts, publication details, etc.) as a little advertisement that will attract readers to their publication, many publishers in the humanities (and other fields) treat citation data as an asset that they can license to services like Thomson and EBSCOhost, who in turn sell access to institutions.  I think it is fair to say that the primary motivation for this way of doing things is not to provide the best support possible for academic research.  (Note that I am not talking about free access to the full text of all articles; just to the basic citation data about each reference.)

Now, you might be thinking that I have confused two issues -- APIs and the licensing of reference data -- and, to some extent I have.  But they are related.  Google provides an API for regular Google searches.  But not for Google Scholar.  Why?  Because the academic publishers would not agree to this because they were afraid of losing their income from licensing their data to resellers.  Thus, we are reduced to trying to keep up with each little change in the HTML that Google makes to their Scholar search results page.

So, back to the original question: why should you care about APIs?  Because the lack of support for open, stable APIs to basic reference data in many fields is holding back everyone  those fields.  We will continue to find ways to make it easier to search the literature in all fields of academic study, but we would be more successful if the publishers actually wanted this to work.

Many of you have, or will have, significant influence on one or more of the academic publishers in your fields.  You may be on the editorial board for a journal.  You decide which journals you submit your manuscripts to.  I think it is time for everyone to use whatever influence they have to push publishers in all academic fields to follow the lead of publishers in the bio-medical sciences.  It is in everyone's best interest (even the publishers) that there be ready access, through both user and programming interfaces, to all of the basic citation data in all academic fields.

The following video demonstrates the use of synchronized libraries on one computer.  The mechanism is exactly the same when the copies are on different computers, but this was the best way to show the updates as they happen.

In the real world, performance will be a bit slower than in the video because you will be further from the servers than we are, but updates will still propagate in just a few seconds.  The biggest difference will be seen when first synchronizing a large library and when propagating PDFs -- which are much larger than typical reference updates -- these will take quite a bit longer than in the video.  But other than when you first create a synchronized library, the volume of data going back-and-forth will be relatively modest, so you should still be pleased with the performance.

Synchronized libraries is working in Sente 6, Preview 3, which was recently distributed to our small group of testers.  We will be making one or more public previews available in the coming weeks, so everyone will be able to kick the tires shortly.

Please let us know what you think -- we want to make sure that our design for synchronized libraries works well for as many people as possible.

Michael

The problem with Dropbox is that it is primarily designed to synchronize files like word processor documents or spreadsheets, not database files.  Using Dropbox to synchronize database files is likely to result in corrupted files.

The problem is certainly not limited to Dropbox.  Any system that copies Sente libraries from one computer to another must only be used when Sente is not running on either computer.  When Sente is running on the source computer, it is always possible that Sente will modify the file as it is being read by the replication software, resulting in an inconsistent copy.  And if you replace a database file on the destination computer while Sente is running, all of the information that Sente has cached in memory will be out of sync with what is in the newly updated file.  In either case, the likelihood of damaging the database is very high.

So, please use mechanisms like Dropbox with Sente only with great care. 

Obviously, we know that synchronizing your Sente libraries across computers is a very high priority for many users and we are exploring mechanisms to do this.  We expect to be able to say more about this in the relatively near future, but until then, please be careful using tools like Dropbox for this purpose.

Michael

Since we first launched Sente back in 2004, our focus has been on providing the best reference manager for the most demanding users.  This has lead to the development of features like standing searches, hierarchical libraries, targeted browsing and automatic PDF import, to name a few.  During this time, the growth in the use of Sente by graduate students, professors and research professionals has been dramatic.

But we have heard from many undergraduates over the years who really like Sente's features and intuitive user interface, but who cannot justify the cost of the software because they will only use it to gather the references for a handful of papers each year.  These users are not really interested in setting up standing searches or importing large collections of PDF files; they just want an alternative to EndNote.

To make Sente more affordable for this type of user, we are now offering Sente Undergraduate Edition, which has all of the features of Sente, but has a limit of 250 references per library.  This limitation, while too restrictive for most graduate students and researchers, is more than enough for most undergraduate papers.

Sente Undergraduate Edition is not limited in any way other than the size of the library that can be created.

Sente Undergraduate Edition is available now from Kagi for $34.95.  The Undergraduate Edition can be upgraded to the regular academic version for $64.95.