Christos submitted Racket with option contracts to the artifact evaluation
committee for OOPSLA after our paper survived the first round of reviewing
w/o a hitch. 

Yesterday we got the AEC's comments back, and Robby and I decided it was
worth sharing them with the very-core developer team. I am including the 
full comments so you can confirm with your own eyes what I am going to 
extract as lessons next. We are mostly forwarding the comments because
we would like everyone to do so when PCs come with AECs in the future (as
long as the AECs are working properly). 

Thoughts: [read between ***s if you don't have time for the whole message]

1. In principle Racket is one of few groups that should be able to 
participate in such evaluations w/o a hitch. We emphasize the idea of 
developing usable software and we look for people who use our software
and we act on their [bug] reports. 

From what we know, about half the accepted papers submitted an artifact, 
and of those 21ish, 3 or so failed to get the seal. My guestimate is that
40% of the papers will carry a seal. 


2. The first two of the AEC reviews are high quality. The reviewers clearly
invested time in reading the paper, working with the artifact, writing down
a useful summary. 


3. We made one huge and obvious mistake. 

In principle the paper is a language design paper. 

But, given that this is OOPSLA, we added performance experiments and they have
the exact same status as theorems in theoretical design papers submitted to 
POPL or ICFP. [We have theorems too.] They confirm that we developed the 
system to the point where the obvious claim/thesis (option contracts reduce 
contract checking overhead) is confirmable. 

Our mistake is that our artifact focuses exclusively on this very minor 
idea in our paper. The AEC reviewers could unpack the bundle, follow the
instructions, and confirm the experiments to any depth wanted: two consume
a fraction of an hour and the large suite consumes a working day. 

That's what they did, and that's bad. 

4. ***********************************************************************

It is obviously bad because we want AEC reviewers to play with 
option contracts and say "I programmed in Racket with option contracts
and it worked". 

This idea was on my mind. I am sure this idea was on Christos's and Robby's mind. 
We failed to act on it. 

What we should have done is: 
-- pointed to the extensive documentation of contracts in the Racket document system 
-- added complete Guide-style documentation for option contracts
-- including a couple of examples that people can copy/paste into Dr and play with 
-- given some suggestions on how to play with them. 
In a sense this is how every single addition to our ecosystem should be done. 

No excuses, we f'd up. 

The AEC reviewers wanted to do more than run experiments, even though the code was, 
in Racket and they couldn't. They looked for docs on contracts and they didn't 
find the Guide. They looked for docs on options, and all they found was stupid 
reference manual documentation. They may generalize and think that we don't write
great documentation for our stuff -- and in general we do, regardless of what
people say about our introductory material. 

Let's do better, much better next time for such papers. 

***********************************************************************

5. Racket did NOT do well either. 

In their search for ways to play with the system, the reviewers seemed to have
read random files, grepped in random directories, and opened our benchmarks. 

As they looked at random Racket files -- probably files that implement option
contracts and the underlying machinery, i.e. follow some random require links -- 
they found Racket implementation files without any/much documentation. 

I know that this is our style and the AECs actually write "we know Racket is 
a self-documententing language" which is either complete total irony, a slap 
in the face of Typed Racket, or perhaps incompetence. 

We need to change our way of writing programs. Please document your
functions. Please follow the style guide and make functions small, readable, 
use documented data, and add signatures.