No decent academically-oriented wiki software should be without a facility for handling citations. For a long time, I’d toyed with, and rejected the idea of building a bibliographic software subsystem into Instiki. Such a subsystem would
One such bibliographic software system is RefBase. Andrew Stacey has recently begun hacking on RefBase, so it occurs to me that one might use RefBase as a bibliographic back-end, and have Instiki grab citations from it, via its XML API.
The purpose of this page is to try to map out what would be required to implement such a scheme.
First of all, we’ll need a new model
class Citation < ActiveRecord::Base
has_and_belongs_to_many :revisions
belongs to :webCitations are associated to Revisions, rather than to Pages, as the list of citations, on a page, can and will change between revisions of that page.
What should the Citation table look like?
def self.up
create_table :citations do |t|
t.string :bibtex_key
t.text :xhtml
t.datetime :created_at
t.datetime :updated_at
t.string :etag
t.integer :web_id
t.integer :refbase_record
end
create_table :citations_revisions :id => false do |t|
t.integer :revision_id
t.integer :citation_id
end
add_index :citations_revisions, [:revision_id, :citation_id], :unique => true
add_index :citations_revisions, :revision_id, :unique => false
add_index :citations, :bibtex_key
add_index :citations, :web_id
add_column :webs, refbase_url, :string
add_column :webs, refbase_username, :string
add_column :webs, refbase_password, :string
endSince we’re using RefBase as a backend, we don’t need to be able to edit, or otherwise manipulate, the bibliographic information. All we need is to be able to do is output it in two different formats
\cite{bibtex_key}, in the LaTeX outputshow output.Question: might we also want to be able to spit out bibtex format, retrieved from RefBase? If so, add another column for that.
We also need to be able to update citations by re-fetching them from RefBase. Hence the etag column. I assume RefBase supports the If-None-Match header. If not, who do I have to beat around the temples?
Of course, we’ll also modify the Revision, Page and Web classes:
class Revision < ActiveRecord::Base
belongs_to :page
has_and_belongs_to_many :citations
composed_of :author, :mapping => [ %w(author name), %w(ip ip) ]
endand
class Page < ActiveRecord::Base
belongs_to :web
has_many :revisions, :order => 'id', :dependent => :destroy
has_many :citations, :through => :revisions
has_many :wiki_references, :order => 'referenced_name'
has_one :current_revision, :class_name => 'Revision', :order => 'id DESC'and
class Web < ActiveRecord::Base
## Associations
has_many :pages, :dependent => :destroy
has_many :wiki_files, :dependent => :destroy
has_many :citations, :dependent => :destroy
has_many :revisions, :through => :pagesWe’ll need a new Controller
class CitationController < ApplicationControllerwith, at least, the usual CRUD actions. The create and update actions will involve querying a RefBase server.
We’ll also need to be able to configure the Citation Controller (supplying a URL for the RefBase server, and perhaps authentication information).
Question: Should that be done in the AdminController or in the `CitationController?
Question: What other actions does the CitationController need? Should we, for instance, be able to obtain a bibtex file of the citations associated to a given page?
We need to support citations in the Chunk Handler.
When do these Refbase lookups happen? Presumably, when the page is rendered. That seems like a real performance hit, but is there any alternative?
This seems interesting, in that regard. But there are probably other, simpler, asynchronous solutions.
How, exactly do we do the lookup? Collect all the citations on the page and query the server in one batch?
The RefBase API seems to require our knowing the RefBase record number. That’s brain-dead. Can we retrieve the data we want, using the bibtex_key instead?
In the above database schema, I assumed we need to keep track of the refbase_record. It would be nice if we could dispense with that.
What do we do when the RefBase server is unavailable, when the given bibtex_key is not found on the server, etc?
So many questions …
Below, Andrew points out that, perhaps, RefBase is ill-suited to the task at hand. He also points out that RefBase may contain stale data.
Perhaps we should simply query the public information servers directly. For instance, consider the example I gave below. In my text, I have a \cite{Weinberg:2008si} (based on the SPIRES Bibtex key).
We could send a query to SPIRES
http://www.slac.stanford.edu/spires/find/hep/xmlpublic?texkey=Weinberg:2008si
to obtain an XML representation of the desired citation. Or we could GET
http://www.slac.stanford.edu/spires/find/hep/wwwbriefbibtex?texkey=Weinberg:2008si
to obtain the Bibtex version (embedded in a <pre> element) of the same information.
Of course, if we don’t know the SPIRES Bibtex key, we could query on the eprint number, instead
http://www.slac.stanford.edu/spires/find/hep/xmlpublic?eprint=arXiv:0810.2831
http://www.slac.stanford.edu/spires/find/hep/wwwbriefbibtex?eprint=arXiv:0810.2831
Unfortunately,
So, if we went that route, we’d need to build an interface for hand-entering, and editing the citations. In other words, we’d need to re-create another bibliographic software package, a road we don’t want to go down.
Andrew Stacey: Firstly, let me say (as I know the RefBase team are aware of this!) that I’m finding RefBase (modulo a few hacks to make it more mathematically friendly) an extremely useful tool, and I’ve not yet explored all its possibilities. However, I’m no longer so sure that it’s quite what is needed here. I originally thought that the idea was simply a link-up system between a reference database and an instiki wiki. Now it seems that a much tighter integration is possible (via Active Resource).
One of my reasons for pulling back is the user authentication for RefBase. It seems that RefBase is designed to be a central server that users log on to and can store personalised information (mainly tags and citation keys, but there are plans afoot to have commentaries as well). This is common to the other database systems that I looked at before settling on RefBase. This seems opposite to the Instiki-style of complete openness (at least on the public webs).
Also, RefBase and the like copy information across from the various sources. One of my hacks was to add a MathSciNet import tool (arXiv was already there). The problem with this is what to do when the source record is updated - which becomes more likely as more preprint servers spring into existence. It’s entirely possible for the RefBase information to be out of sync with the arXiv (say) information. Automatically updating is problematic since other information may have been added by the user.
On the other hand, one does want to allow for some local storage since it may be useful to point to people’s homepages where it is less sure that information will stay.
I’m trying to imagine what would be a useful system for the user. There are two places that I can see a bibliographic system coming into use: as a source of citations, and as a “thing to be discussed”.
As a source of citations, I guess we want the user to put some code like \cite{key} and have the Instiki process ask the database process for the record associated with key. The problem with that is that the user has to go across to the database to get the correct value for the key anyway (can anyone remember all their bibtex citation keys?), so why not just have the reference program spit out the right markdown syntax to copy-and-paste in? One could easily just use footnotes for this. Much better would be if the user could put in a search term instead of the key and have the reference system offer a list of possible matches from which the user could select the right one. That needs a couple more passes of the data, I suppose, but would be way more useful.
As a reader of the page, I want to be able to click on the citation and be taken to the page for that citation, from which I can, in a click or two, go to a variety of bits of information for that reference. Here’s where it’s most like the current systems and doesn’t need too much integration.
These are just my initial thoughts. No doubt they’ll change as I think a bit more about it.
Jacques responds: One of the original motivations for my work on Instiki was to produce an environment where people could bat around ideas which might eventually work their way into a paper. So the LaTeX export function is important. I’d like to be able to spit out a LaTeX version of a page, paste that into a paper I am writing, and have it “just work”. In that version, citations look like \cite{Weinberg:2008si}. But, on the wiki, I want a hyperlink to a formatted reference at the bottom of the page, which looks like
Steven Weinberg, “Non-Gaussian Correlations Outside the Horizon II: The General Case,” Phys. Rev. D79, 043504, arXiv:0810.2831 [hep-ph].
I realize that this use-case may not accord very well with the use-cases that RefBase was designed for. But I don’t think it’s too far off the mark.
This seems opposite to the Instiki-style of complete openness (at least on the public webs).
Not irremediably. We’d just need to create an “Instiki” user on the RefBase server (or, for finer control, one such user for each web on our Instiki installation).
I also agree that building an interface to RefBase’s search facility would be a nice 2nd step. But first things first: being able to extract data from RefBase, store it in Instiki, and spit out a formatted version on-demand, would be quite a nice first step.
Andrew Stacey It’s tempting to use RefBase. It’s there, I’ve already a little experience in hacking it (though I should make clear that most of my hacks are “surface level”, I’ve not had to look too deeply into the code), and it almost does what is wanted. I picked it because it was almost what I wanted, but I then found that it wasn’t quite what I wanted and so started hacking it (if the RefBase developers are keeping an eye on this, I have a half-written email to you guys describing my hacks and asking if you want me to send any to you. One day I’ll finish the email …). Of the various programs I tried, it seemed the easiest to hack, however that wasn’t because it had lots of plugins and modules, but because the code was well commented.
But my point is that it probably isn’t quite right, so to get something right for Instiki then it’s going to need a little tweaking. So I’m wondering whether or not it’s best to work out exactly how this thing would be used and then see if RefBase (or other) is close to what is wanted.
So here’s a workflow from the writer’s point of view.
\cite{key} at the appropriate juncture.Here’s a workflow from the reader’s point of view.
Okay, so the Instiki+RefBase/Whatever process needs to do the following:
It feels as though you’re concentrating on the 3rd step here.
But let’s take that and run with it - I’m happy to work on several levels here. And let’s stick with RefBase since I have a working installation that I’m happy to play around with and I already have a little familiarity with the code.
One could easily subvert the authentication. If the installation was dedicated to the relevant Instiki process then you’d want it open, just with good logging. So then you want to send RefBase a list of references and get a nicely formed XML back. RefBase certainly can export XML so this should be no problem. Then Instiki can cache the XML, updating it if the citations on the page change (though it may be good to have a way of manually forcing a reload).
Either RefBase can itself export the bibtex, or Instiki can internalise the conversion. Doing it internally would make it easier to change the RefBase component for something else, but as you say right at the top that would involve writing a Ruby BibTeX library. On the other hand, one could simply use the bibtool program (which is what the main branch of RefBase does anyway).
Ah, I’ve just noticed something in what you said above. You want to be able to ‘cut and paste’ from an Instiki TeX export into another paper. That means that you’ll need your citation keys to match, and what you choose and what I choose won’t necessarily be the same unless we all use the same RefBase installation as our source for references. That is, the paper that you’re writing must also get its reference data from that RefBase installation. That complicates matters a little, either way.
Suppose we’re writing a joint paper. We’re not doing the whole thing on Instiki, but parts get developed there (maybe I’m being a bit pigheaded about using \( instead of dollars) and others bits are developed locally. I figure I’ll get in a bit of self-promotion and flagrantly cite ‘Comparative Smootheology’. As it’s a paper I wrote, it’s been in my reference database since before anything appeared in public and I have a citation key from that time. So I cite it \cite{as8}. You, in a gesture of generosity, meanwhile decide that you’ll help me with my citation index and cite it in the part you’re developing offline. So you cite it as \cite{math/0802.2225}. Meanwhile, over in the public area we also include it and cite it as \cite{782} (since that’s it’s key in the database). At some point, this lot needs sorting out.
The easiest way is if we’ve agreed to use the RefBase installation from the start. But that means that I need to be able to put up all sorts of crud that no-one else is particularly interested in because if I’m using it for one of my papers then I’m going to use it for all of them otherwise it’s yet another system to learn.
The more complicated way is for someone to manually sort out the various references. I guess the easiest way is for each program to export its references in BibTeX format, compile the document, and then look for duplicates.
The most elegant way would be if the reference program was really just a portal and that when I type \cite{as8}, you type \cite{math/0802.2225}, and we both type \cite{782} then each reference program simply converts that to a citation to the arXiv reference and the BibTeX version of the arXiv reference is extracted. This would involve a layer on top of the usual \cite command: there would need to be a “look-up list” so that several keys could point to the same citation. It’d be a pain to do in TeX, but not difficult (I’ve a little experience of hacking TeX to do associative arrays and object-oriented structures; I probably didn’t find the best way of doing it, but I found ways that worked).
Again, I’m just trying to think of the right design before integrating the system too closely with RefBase.
My other caution on RefBase - which is more about it being PHP than anything else - is the difficulty of having two obviously separate systems. It appears that some still have issues with RSS and won’t notice an additional system like this unless it is grafted into Instiki. That would be easier if it were a Rails app.
Andrew Stacey (5 minutes later). I hadn’t spotted your added section above. Okay, so you don’t want to write your own reference software. Fair enough, I’m not too keen on it either! RefBase doesn’t currently do SPIRES, but it shouldn’t be hard to add (I added MathSciNet which just goes to show how easy it is to do). The stale data is a problem, but then it’s a problem that RefBase should think about anyway. That makes me think that there are two classes of hacks that we’d need to make to RefBase: general improvements, and specific integration issues. I’ve no hesitation about doing the first. What I’m a little hesitant at is doing the second. So what I’m trying to figure out is how much of the second type will there be? Will we look back in 6 month’s time and say “It would have been easier just to write our own app.”? I hope not, but to get a fair idea then I think it’s good to play through the possible usage scenarios.
Andrew wrote:
- Make it easy to find the relevant cite keys.
- Make it easy to create a useful relevant page.
- Produce citations in the relevant format for inclusion in the corresponding output.
It feels as though you’re concentrating on the 3rd step here.
Step 2. is the raison-d’etre of Instiki. Step 1. is (or should be) the raison-d’etre of any bibliographic software system (of which RefBase is the proximate example). That leaves Step 3…
I don’t really know whether RefBase is the “right” bibliographic software system to be working with. Indeed, if you look at most of what I’ve written above, it’s essentially independent of the bibliographic software system being used (largely, but not entirely, due to the vagueness of what I’ve written).
If there’s a better choice than RefBase, let’s use that. I agree that a Rails App would be easier:
Unfortunately, there isn’t — to my knowledge — an existing Rails bibliographic software package to work with. We could write one from scratch. It might even be fun. But would it be the best use of our time?
The rest of the problems you cite (our using different bibliography managers, different citation keys, etc …) are very real. But I don’t expect to solve them. Let’s even forget (for the moment) about the whole paper-writing enterprise. Let’s just ask if we can build something that would streamline adding citations to an Instiki wiki (like the nlab).
Not sure of refbase is the right choice for you, but there was some somewhat poor information above, so I thought I’d clarify some things:
Browser Cruft, Installing on Shared Hosts, Installing under MacOSX Tiger