Som bakgrunn for sitt foredrag har Erik Sandewall oppgitt følgende artikkel:
Electronic Transactions on Artificial Intelligence, ETAI
Vol. 1 (1997): 1-12
http://www.ep.liu.se/ej/etai/1997/001/

Editorial Note

Publishing and Reviewing in the ETAI

Erik Sandewall

On 20.11.1997, the Electronic Transactions on Artificial Intelligence (ETAI) received the third and final referee report for the first article that had been submitted to it, "The Complexity of the Language A" by Paolo Liberatore of the University of Rome "La Sapienza" . The article was accepted since all three referees recommended so, making it also the first article to be accepted to the ETAI.
This decision, otherwise a routine situation in our lives, is of interest because of the sequence of events that led up to it. The ETAI does not use the traditional procedure of confidential reviewing before publication, and has replaced it by another procedure which we believe is better able to guarantee the quality of publications, and to facilitate communication between researchers in our field. It is therefore appropriate, at this time, to describe this sequence of events in the case of Liberatore's article, and also to relate it to the original intentions when the ETAI was started half a year ago. As you might expect when a new system is tried out, we have had to improvise at some points, and at other points we have modified our procedure somewhat in view of new insights. Reviewing what has happened puts all of this into perspective, and it is also a good way of explaining in very concrete terms how the ETAI operates.

1 First Publication

After preliminary contacts to find out about the ETAI procedure, the author delivered his finished article on June 30, 1997 in the form of a postscript file generated from Latex with a standardized style. Four days later, it was published by the Linköping University Electronic Press (LinEP), so 4.7.1997 is its correct date of publication. (If we are going to be serious about giving proper credit to the first author who publishes a particular result, then we must be precise about when each article was published).
In the ETAI system, "publication" means exactly what it means in ordinary language or to a lawyer: the day when the article was made public. The ETAI procedure is as follows: first the article is published, then it is reviewed (= discussed publicly), then finally it is refereed (= a confidential pass-fail decision whether to accept it or not). More about this and about a few intermediate steps later on in this text.

Publication by an electronic press such as LinEP involves several things:

One more important thing: the e-press does not promote or "market" the article. It is left to the author or to third parties to make the article known to the world. In this sense the e-press is analogous to a printing house rather than to a publisher, which anyway corresponds to the original meaning of "press" in this context.
The ETAI does not deal with manuscripts; it only deals with articles which have already been published electronically by an organization such as LinEP. Note that it is not sufficient that the article has been "posted on the net", for example if the author has put a link to it from his1 individual homepage. The ETAI only recognizes articles that have been published by organizations that perform the now-mentioned functions in a reliable fashion (or, exceptionally, by a combination of widely accessible paper publication and less persistent posting). The rationale is, of course, that if our scientific community is going to invest work in reading, reviewing, commenting, refereeing, and later on citing an article, then we want to be sure that the article stays and is a reliable reference; otherwise the invested work would be wasted, and the citations would be meaningless.
For the same reasons, it is not advisable to operate an e-press on a departmental basis. It is difficult to guarantee security if it is so close to the authors. The Linköping E-Press is operated by a cooperation between the university's computing service and the university library.
According to the blueprints, participating universities are expected to have their own e-presses in order to serve their own departments and research groups. However, while we wait for that to happen, existing e-presses have to help out even for researchers from elsewhere. Thus, although Paolo Liberatore is at the University of Rome, his article was published by Linköping E-Press. As mentioned, this could be arranged without noticable delay.

2 Informing the Peers and Obtaining Feed-back

Just publishing an article is not meaningful unless it is also brought to the attention of the peers, that is, other researchers who work on similar problems. These are the people who compete for being the first to find solutions for commonly recognized problems, and who are also the most likely ones to read the article, to comment on it (with appreciation or critique), to use it in their own work, and to cite it.
ETAI's strategy is to create a discussion around every received article. In order to characterize what the discussion can be like, we appeal to the common experience of a question period after every paper presentation at a conference: there is time for questions from the audience and answers from the author. Sometimes, these are questions for clarification, but at other times they may reflect genuine disagreement and may extend into a more general debate.
Before the advent of the Internet, conferences were the only possible way of having such interactions. Now, we can do them on-line, and over a period of three months instead of five minutes. This has several advantages:

The original plans for the ETAI foresaw the use of a web page for this kind of interaction, and a News Journal that would be issued monthly and summarize the discussion during the past month. Those who were actively interested in the article would check the web page from time to time and write contributions; others would receive the news journal (or a reminder of the existence of a new news journal issue) allowing them to catch up.
Since Liberatore's article was the first one to be submitted, it was also the first one where this scheme was tried. The outcome was only partly encouraging: we did get a number of interactions, but not as many as we had hoped. The number of visits to the web page in question was also below expectations. This may partly have depended on the season, since the July-August period is not the most active one. Anyway, for the following articles we decided to modify the system so that debate contributions are broadcast by e-mail on the same day as they arrive, if possible, and in an edited Newsletter with a uniform appearance. This appears to lead to considerably increased activity. For the most active article, by Antonis Kakas and Rob Miller, we had 13 interactions and a spin-off into a broader discussion already within one month from the publication date.
In this context, we may return to the question of the concept of "publication". Some of our colleagues insist that that term should only be applied to publication in a journal (or quality conference proceedings), subsequent to confidential peer review. We observe, however, that the world has changed considerably since the contemporary peer-review system was created. Fifty years ago, when typesetting in lead was the only way of preparing multiple copies of an article (besides using a typewriter and carbon paper), it was natural to review an article before publishing it, simply in order to decide whether it was worth the expenses involved. But now that articles are anyway distributed very freely and at unnoticable cost, using the Internet and otherwise, the old terminology does not make sense. How can you say that an article is "unpublished" when it is literally available to the whole world by the click of a mouse? Formally treating it as published is appropriate and necessary in order to safeguard the authors' priority rights to their results (which is not the same thing as their copyright), and to protect them against all the delays and other surprises that can happen in the conventional review process.

3 The Summary

Authors of ETAI submitted papers are encouraged to write a summary of their article. This is another one of ETAI's innovations. The idea is simple: In the ordinary A.I. literature, there is a very big difference between the abstract and the full article, in particular since A.I. has developed a practice of allowing quite long articles. Reading such an article takes considerable time. On the other hand, abstracts are too often limited to phrases such as "we propose a new approach to performing X and describe its advantages over previously proposed approaches". In the terms of bibliographers, these abstracts are indicative rather than informative: they may facilitate classification of the article and they may wet the reader's appetite, but they do not help at all for identifying what the results are.
At the same time, readers would be very well served by informative abstracts. Given the large volume of articles that are published today, noone is able to read everything. One way out is to focus on a few "prestigious" periodicals and to pretend that everything else does not exist. This attitude is widespread but unfortunate, and if as a field we are going to become more serious about respecting first published results regardless of where they were published, we must make it easier for authors to identify relevant articles in a large number of publications.
This is where the summary is useful. It should be intermediate, in size and in precision, between the indicative abstract and the full article. With a recommended length of 2 1 pages, where each page contains the same amount of text as the pages of the present article, it is possible to specify the concrete result of the article in that summary. Besides being helpful for readers, this also serves as a check on the article itself: if it does not contain any result that can be stated in two pages, then maybe it should not be published (or read) at all.
In the case of Liberatore's article, the summary was returned on July 8, and immediately posted on the article's interaction page on the web.

4 After the Reviewing Period: Option to Revise

According to the ETAI procedure, the reviewing period shall last for three months, and after that time the author is asked whether he wishes to submit the article to ETAI refereeing, and whether he wishes to revise it first. The idea is that the feedback that has been obtained during the discussion period shall be used to improve the article. Also, with this system, the author is free to submit the article to another journal if he prefers to do so. This means that it is quite possible to let an article participate in ETAI reviewing (the open discussion) while it is being considered for a conference, since this does not constitute double submission. The only requirement is that submitting it to ETAI or another journal must not be done before the conference reviewing has been completed.
Paolo Liberatore answered at once that he wished to submit the article, since this had been his intention all along. Also, the interactions during the discussion period had only consisted of clarification questions in his case, so they had not given any reasons to change the article. However, meanwhile he had obtained an additional result which could be seen as an amedment to the article. Was it appropriate to extend the article with that new result?
This is another point where the ETAI scheme differs from traditional journals. In the ordinary journal, at least in computer science, the editor might well have agreed to this improvement of the article, even if it were proposed after refereeing. In the case of ETAI, this would not be appropriate, since the new result would then be exempt from a discussion period, and since the date-stamp on the results would become misleading. Therefore, the author was advised to write up the amendment as a research note with a clear tie to the original article. The research note obtains its own date-stamp, it obtains its own, three-month review period and its own refereeing, and it can be written so as to reuse the main article's motivations, definitions, and reference list.
In a conventional paper journal, such a separation between the first article and the amendment would be somewhat inconvenient for the reader, who is then forced to move between issues and volumes of paper in order to cross between the article and the amendment. In the electronic medium, the same step is done by the click of a mouse, and it is then much less of a problem.
For these reasons, it was the original article which had been published on July 4 that was sent to the referees on October 21. The stipulated discussion period of three months had been extended by two weeks in compensation for the summer lull.

5 Refereeing

This article was sent to three referees: two members of the editorial committee for the ETAI area in question, and one external referee with special competence in the area of the article. (The ETAI does not have one big editorial board; instead, it is organized like a federation, with an area editor and an editorial committee of limited size for each of these areas). The referees were asked to answer the following questions within two weeks:
  1. Are the results of the article, as specified in the summary, of significant interest for researchers in our area? In other words, are these results which everyone who does active research on reasoning about actions and change should know about? (Answer Yes or No).

  2. Does the full text of the article substantiate the claims made in the summary? ("Does it deliver what the summary promises?")

    (Answer Yes, No, or "No, except..." where the exceptions should specify some parts of the paper which you have not been able to verify. For example, if there is a lengthy proof of a very technical nature which you don't want to wade through, then specify that in the except clause. We will then encourage the readership to do the verification, and publish a positive (or negative) check as a separate entity)

  3. Is the text intelligible and concise? (Answer No if it is confused, undefined terms are used, overly verbose, or overly brief; otherwise Yes. Do not bother with minor corrections of style, etc).

  4. Do you know of any previous publication of the same result, or has such a previous publication been convincingly reported during the open discussion?

  5. Are there any other considerations which may affect the decision of acceptance? (The default is that every article which obtains Yes'es for questions 1-3 and No for question 4 will be accepted).

We have intentionally chosen these questions which have an operative flavor, and avoided questions such as "please grade the scientific quality of the article on the following 10-point scale from excellent to poor" or "does the article contain adequate references to relevant earlier work?" It seems to us that the notion of "scientific quality" is used very differently by different reviewers, and quite often it is the vehicle for rejecting work whose approach one does not like.
References to earlier work is a very important point. In traditional journal reviewing at its best, reviewers check that earlier work is correctly referenced, and if this is not the case the article is rejected. This ideal does not work in A.I. today, and maybe it is too much to demand from authors and reviewers given the pace and volume of new publications. Sometimes, if a reviewer points out the failure to reference a particular earlier work by X, one is tempted to infer by abduction that X is the reviewer.
Rather than trying to fix the classical method for assuring the coherence of the literature and the respect for priority of results, we must therefore look for new approaches. The open discussion during ETAI's review period offers such a possibility: it becomes possible and socially acceptable for anyone to point out a missing reference. In a confidential journal review, citation complaints are difficult to deal with and full of overtones because of the secrecy and the lack of dialogue. In the ETAI discussion it is much easier: the comment is open, the author has a chance to answer, and if the commenter and the author don't agree then the rest of the world can make up their mind anyway. Most important of all: the issue about the missing citation does not in any way influence the author's priority to his or her results. This is at it should be, since the result is presumably the same regardless of what is included in the article's bibliography.
The ETAI questionnaire to referees is simple and concise, and this should facilitate work for the referee and allow her to return the answers more rapidly than otherwise. In the present case, this worked as intended, although one reviewer requested right from the start to have one extra week, which was granted.
According to the policy, ETAI referees are asked not to make detailed comments about the article. The instruction was: if you observe things in the article that ought to be changed, but which do not affect the pass/fail decision, then it's too late now; you should have thought of that during the review period. This idea did not work, since some referee(s) anyway discovered certain minor flaws in the article, ranging from a few minor spelling mistakes to suggestions of how some proofs might be made easier to follow.
For the record, it is important to state that none of these suggestions indicated any problem with the article. Small details like this occur for everyone, and in this case there was just a small number of them. What I did as area editor, at this point, was to sort these comments into three categories:

Comments in the first group were only sent to the author, for his feed-back; they were not of interest to anyone else. Those in the second group were posted on a separate page that is attached to the article's interaction page; the idea is that these comments are not of wide interest in the sense that they ought to be circulated by Newsletter or News Journal, but they ought to be available for anyone who wishes to read the full article carefully. The comments in the third group were posted as questions to the author in the general discussion session about the article, with "Anonymous Reviewer" as the originator of the question. At the same time, the author was invited to also use the clarification page, both for putting his answers in, and for adding similar comments that he receives by other channels.
Anyway, all three referees agreed that the answer to the first three questions was Yes, and the answer to the last two questions was No. One of the referees quoted an exception for question 2; this exception was resolved in dialogue with the other two referees.

6 How Finished Should a Journal Article be?

From a classical vantage point, the procedure described here may seem unsatisfactory; one may object that the accepted article is not really finished. An amendment is attached as a separate entity instead of having been "properly" inserted into the article; some of the possible corrections are listed on separate web pages and the original article is still offered to the world; there is not even a requirement to fix the spelling errors. Is this consistent with the concept of quality publication of research results?
Our answer has two parts. First, and most importantly: the character of research in our area is such that no article is ever finished, even if it has been published in the A.I. Journal. There will always be objections, reports of counterexamples, further developments, missing citations, and so on. All this information is semantically attached to the article. It is much more important to find ways whereby those semantical attachments can be made explicit and easy to follow, than to give final polish to an article that pretends to be complete and self-contained although in fact it stops being so even before it comes out of the presses.
The ETAI tries to provide those attachments. The article interaction page is not discarded on the day the article is accepted or rejected by the referees. On the contrary, it contains valuable information for future readers, since when you read an article it is very good to know what others thought about it, what questions they asked, and what the author answered. After all, that is one of the reasons you go to seminars and conferences. There is no reason why one cannot pose additional questions later on, either because someone thought of some additional point, or even for asking how this article and this author would view another result that is published later than the article at hand.
Both the traditional media for communicating scientific information - journals and conferences - embed the presented articles in various types of surrounding information: reviews, citations and cross-references, letters to the editor, questions to the speaker, and so on. The Internet and WWW technologies allow an enormous increase in the size and depth of that surrounding information, and much more rapid interactions. When we make full use of those possibilities, as in the ETAI, traditional values such as having large, self-contained articles which have been polished to stylistic perfection become less important. It is the contents and the timeliness that count the most.
Let it be quite clear that we do not argue for a decrease of quality, but for a different emphasis and for an increase in some aspects of publication quality compared to what is the case today. In particular, the quality of the literature in a field of research is not merely the sum or the maximum of the quality of the individual articles in it. The inter-article coherence, the attention to earlier work and to approaches other than one's own, the systematic evaluation of results and not only the repeated presentation of (hopefully) new approaches - all of these are also important aspects of scientific quality.
In summary, then, the outcome of the reviewing, revision, and refereeing processes in the case of Liberatore's article were quite in line with ETAI's style. They resulted in an article, a research note, and a structure of other information which continues to be available on-line for everyone who reads or considers reading the original article.

7 Post-Refereeing Corrections of the Article

This said, it still remains that there is no reason to leave minor mistakes and possible improvements unattended; we just need to make sure that other and more important ETAI principles are not violated in the process. The concrete question, in the case of Liberatore's article was: the article has been published, then discussed, then refereed as is, and now we have some suggestions for minor changes. What do we do?
The decision was to encourage the author to produce a new version of the article but not of the summary, and to limit the changes to those that regard the presentation - no changes that improve the results. In particular, for theoretical articles it is not allowed to change the wording or content of the theorems, which after all represent the results in maximally condensed form. The new version of the article is published by the Linköping E-Press just like the original version, and both versions are referenced from the same E-Press entry page. Both are timestamped, and the date of issue of each is clearly marked.
The decision to keep the summary fixed, but to encourage changes of presentation in the article that substaniates the summary, is consistent with how the summary is otherwise used as a key to the new results presented in the article. The convenience for the readers of such a system has already been mentioned, and likewise the role played by the summary for the concise questions that are given to the referees.
There have been some proposals to take this even longer, and that for the definition of priority of research results, we should only consider those results that are stated in the summary; if an author had a result somewhere in a corner of an old article, but never mentioned it in the summary, then presumably he didn't realize its importance, so it ought not to count. This is an extreme view, and it is not clear that it would work, but maybe we should go in that direction. By comparison, the view that articles don't count unless they have been published in one of a small number of periodicals is much more bizarre. One way or the other, we have to cope with the large number of publications.
In such a context, it would make a lot of sense to require the summary to be fixed from the date of first publication. This allows one to apply or to approximate the following simple model: The article may be improved later on, proofs may be made easier to follow, citations may be added or even removed, but the summary stays. If a result is not stated in the summary, it does not count. If your summary makes a claim that is not delivered by the body of the article, then the referees will answer No to question number 2. If you want to extend the summary, then you obtain a new and later publication date.

8 Defining a Paper Edition of the ETAI in Terms of Issues and Volumes

As we all know, traditional journals are published in terms of issues and volumes. Several articles were grouped into one issue in order to be printed and mailed together; several issues were grouped into one volume in order to standardize which issues would be bound together by the bookbinder in a labor-expensive operation that is done separately for each library. Consecutive page numbering was used throughout a volume so that the bound volume would look like a "real" book. One may view this as merely a convenience measure, namely if one feels that the use of a journal in archive mode is more important than its use on arrival. Alternatively, one may say that when journals were new on the scene, they tried to emulate the older "book" technology.
For an electronic publication medium such as the ETAI, there is a similar tradeoff. Should the appearance of ETAI accepted articles be oriented towards how they are used on arrival, which now happens via the Internet rather than the post office? Or should they take the "issue" situation into account, that is, the batch of articles that a reader may wish to glance through at one time? Or again, how shall we accomodate the archival situation, where the accumulating body of all accepted contributions to the ETAI are to be preserved in an organized manner?
The ETAI Policy Committee has decided that the ETAI ought to have a paper edition that is organized in terms of issues and volumes, just like conventional journals. The reasons are both psychological and practical: psychological, because the community may not feel at home with a medium that fails to use these well-known notions, and practical, because we can not assume that these familiar structures only serve the technical needs of the paperbound bookprinting technology. If they serve no other need, then they will whither away, but it would be premature to discard them at once.
Being the first accepted article in the ETAI, Paolo Liberatore's article will also be the first one to go into the paper edition. The concept of such a paper edition raises a number of issues, both formal, technical, and policy issues.

First of all, whereas traditional journals claim that they "only consider previously unpublished articles", the ETAI has the policy of only considering articles that have been published (without refereeing) by an e-press type organization. If the article is first published and then refereed, what about the edition that contains the accepted articles? Put in this way, the answer is obvious: it republishes those articles. That is all there is to it. The paper edition of the ETAI is a periodical that appears with one issue every three months; it republishes articles that have previously been published by e-presses, and which have since been reviewed, refereed, and accepted. In fact, besides the articles themselves, the paper edition also includes a protocol of the reviewing and refereeing process, as shown for the Liberatore article later on in this issue.

The fact that this is a paper edition does not of course preclude putting the same information on-line on the net. It just means that one way of obtaining the paper edition is to download it to one's printer. It is a paper edition, no matter if it is transmitted to you by truck or by Internet.

The order of articles in an ETAI issue is determined by the order in which they were accepted. Therefore, issues and volumes can be defined incrementally: as soon as an article is accepted, it is appended to the current issue, and obtains e.g. its proper page numbers. At the end of the accumulation period, the issue is closed, and the next issue is opened for receiving forthcoming articles. This scheme also makes it possible to always keep a partial issue on-line even during the accumulation period.

There is one point where we differ from traditional journals: in our case, each issue is defined for a three-month period, and each accepted article is put into the issue and the volume for the period where it was first published. The time when it was accepted is irrelevant. For example, Paolo Liberatore's article goes into the issue for July - September, since it was first published on 4.7, and although it was accepted on 20.11.

It is natural to use the same page numbering scheme as in other journals, that is, to use consecutive numbering throughout the year. This causes a technical problem. As a consequence of this policy, the full text of an article has been completed before it was known whether it needed any ETAI page numbers at all. Also, a new run of the article through Latex in order to change the page numbers will invalidate any scheme for safeguarding the integrity of articles. How then do we obtain consecutive page numbering throughout one ETAI volume?

The present solution to this practical problem involves operating on the article files on the level of the postscript code. The reader is invited to take a look at Liberatore's article which follows the present one in issue 1-3, volume 1 of the ETAI: the ETAI version of his article has been derived from the e-press version by running a postscript-editing program that removes the first three pages, renumbers the following ones, positions the numbers symmetrically on odd and even numbered pages, and adds a blank page at the end if necessary. Not very high-tech, but it works.

9 Publication and Distribution of the Paper Edition

Conventional journals are financed by sales: scientists and libraries pay for subscriptions. We all know that the cost of those subscriptions is becoming an increasingly taxing burden on the scientific community, although they are necessary for its internal communication of results.
In the case of ETAI, there are no subscription fees. That does not of course mean that the whole process happens without costs; it is just that most of those costs are assumed in a natural way by the author side and the reader side. The author side arranges for the first publication of the article using an e-press; the reader side has the costs of computing equipment, network access, and local printout. However, most of those costs tend to be fixed rather than marginal ones in this context. The intermediate layer between the author side and the reader side has been minimized and can be done as a community service, just like the work in editorial boards and program committees is done already.
The electronic preparation of the paper edition will be done within the same framework. Generating the administrative pages is a minor issue; reconfiguring the articles so that they have the right page numering has been automated. Therefore, every user is free to make his own do-it-yourself copy of the paper edition. All one has to do is to print out the required pages, two-sided if the printer allows it, staple or glue the pages together, and one has one's own instance of the issue or volume in question. There is a page for the cover, which looks the nicest if you print it on a color printer (they are not very expensive these days), and the cover has marks suggesting where to cut in order to get from standard A4 size to a ETAI's characteristic size and shape. We do not foresee any need to impose access charges for this.
However, some users will presumably wish to receive paper copies of issues or annual volumes on a subscription basis without having to be troubled with the printing. We are presently negotiating about arrangements for this, with the assumption that it can all be done at self-cost. There is no need to use incomes from the sales of the paper edition to finance the ETAI activities. This means that the cost can be held quite low.
It also has another important effect: unlike ordinary journals, we are not constrained to have a certain number of pages per issue, or per volume. Under a subscription scheme, if you have too many pages per volume you overrun the budget, and if you have too few, your subscribers feel cheated. This has the obvious consequences on the editorial policy of such a journal: temptations to accept papers which are below standards, especially in the beginning of the journal's life; pressure on the authors to reduce the length of articles even when the contents would merit a longer text, and a backlog of articles which are waiting to be published.
In the ETAI, none of that happens. Articles go into the issue for the quarter-year of their first publication, as already explained, and issues can be as thin or as thick as the situation requires.

10 A Third Form of Scientific Communication

The ETAI has only operated for half a year, so it is still too early to draw any definite conclusions. However, based on the developments that have already taken place, including a range of additional activities that have not been described here (on-line panel discussions, colloquia, bibliographic services, etc), I think that we can already discern a fascinating new trend, namely the development of a third form of scientific communication. We have two traditional means of communication within science: journals and conferences. (I do not only mean conference proceedings, I mean the entire conferences as events). Journals, in particular, are supposed to have evolved from the earlier practice, centuries ago, where scientists exchanged letters between each other. At that time, it could take several weeks or even months to send a letter from one European university to another. Our journals still operate at that pace, or worse. The ETAI is an example of a new forum of scientific communication which inherits some of the characteristics of both journals and conferences - notice how the traditional question period in a conference session has been inherited and extended in the ETAI! However, the speed, the depth, and in general the conditions of dialogue that are being achieved in the ETAI discussions are orders of magnitude better than what would be possible in either journals or conferences. In this sense, activities such as ETAI promise to be a third medium of scientific communication, better in some ways than both of its predecessors. This may well turn out to be the most significant aspect of the evolution of ETAI during the last half-year.

Footnote:

  1. Apologies for using the masculine pronouns generically here and in the sequel.

Erik Sandewall
Editor-in-chief
Linköping University Electronic Press,
Linköpings Universitet,
58183 Linköping
Sweden
erisa@ida.liu.se

Nordisk konferanse om elektronisk publisering
Programme in English