## Contributions to the BIG-bench Manuscript

We are now soliciting contributions to the BIG-bench manuscript from task authors.  You can see the current latest draft status [here](https://github.com/google/BIG-bench/tree/main/docs/paper/BIG-bench.tex). To enable external authors to contribute analysis, we have released data files containing the performance of all models on all tasks, as well as an [analysis colab](https://github.com/google/BIG-bench/tree/main/notebooks/scaling_plots.ipynb) that generates plots like those in the paper. We have additionally released complete logs BIG-G and BIG-G sparse models interacting with tasks. These are stored in the results folder within each task folder (you will need to install [git-lfs](https://git-lfs.github.com/) and use `lrzuntar` to extract the large log files). Because of the extensive potential author list, we are incorporating potential changes to the manuscript with a process similar to the task submission pipeline, though it will be much more lightweight.

Authors that would like to contribute changes to the draft should open a PR with those changes applied to the most recent draft of the paper.  For an example, see this [example PR](https://github.com/google/BIG-bench/pull/710). Add the *draft edit submission* label to your PR. We will review PRs until April 24, 2022.
PRs will undergo lightweight review from the core team, primarily to ensure submission quality.  The review criteria are:
* The PR does not introduce any breaking changes to the main LaTeX document.
* That large blocks of new content are added by creating a new *.tex* file in the contributed folder. This can be incorporated into the body by inserting a `\input{[new file]}` command in *BIG-bench.tex*. (see the way in which *cause_and_effect_analysis.tex* is incorporated)
* The author's PR does not make extraneous changes to the manuscript (e.g. no changes to text justification for unrelated text).
* That the author's PR is not a duplication of work already contained in the manuscript (with a more stringent notion of "duplication" for the text in the main body vs. the appendices).

For the purposes of tracking, if you plan to submit a PR, please open a GitHub issue briefly describing your intended change so that other authors can more easily check that they are 1) not duplicating work or 2) contact you to collaborate.  Add this as soon as possible so that we can more effectively mitigate duplicate efforts. For this reason, please also open a draft PR with your edits as soon as possible, before they are complete.

Below are proposed analyses that we have not done ourselves, but expect would be interesting to carry out and add to the draft.  If you plan to perform one of these analyses, please let us know, and add your intent to the above spreadsheet so that we may connect you with other interested authors.

* **Analysis Proposal 1:**
Sort the BIG-bench tasks by
[average performance of GPT models] - [average performance of Google models]
[average performance of sparse models] - [average performance of dense models]
Print out the tasks at the beginning and end of each sorted list
Try to identify themes for types of tasks that sparse or dense models are better at, or Google or OpenAI models are better at.
are sparse models good at tasks requiring memorization?
If we can't identify themes, put the sorted lists in an appendix, and use it as even more evidence in the text body for the surprising similarity between model classes

* **Analysis Proposal 2:**
Find tasks where different subtasks require different numbers of sequential logical steps to perform. There are several of these.
plot performance as a function of scale for 0-step, 1-step, 2-step, ... subtasks
is there a clear signal in scale, where 0-step tasks breakthrough, then 1-step tasks, then 2-step tasks...?

* **Analysis Proposal 3:**
Find tasks where "shot"-ness causes performance to become worse with increasing numbers of shots (if any), and try to find any commonality between such tasks.

* **Analysis Proposal 4:** Highlight task brittleness -- for instance, as evidenced in tasks where appending choices to input strongly effects performance.
Are there tasks where performance is strongly metric dependent, or output regex dependent?