Software:

• GPU Access: Google Colab provides free GPU usage up to 12 hours/day for academic purposes. We will also try our best to satisfy individual needs through discussion.
• Pytorch Tutorials: These tutorials do a good job of introducing Pytorch.
• AllenNLP: AllenNLP is a Open-source NLP research library, built on PyTorch. It is widely used in the community now and it will be useful to make yourself familiar with it.

Project 2 Final Report Guidelines:

The final report should be 6 pages in 11pt font. Try to make it look like a published article. You can download the latex format files from here (or any other conference).
Below are some guidelines on how to structure your report (these are just guidelines, feel free to restructure to suit your project’s needs; also, don’t use the title below as your section titles; choose what’s appropriate to your work)

• Introduction : What is the problem? Why is it important? What is your approach? What is the goal of your paper?
  • 1st paragraph: what is the problem/task you are addressing in this paper? Why is it interesting; give some examples of applications if possible.
  • 2nd paragraph: How have people attempted to solve it?
  • 3rd paragraph: What are the limitations of past approaches? (These are the limitations that you will try to address and improve upon in this paper). How important is it to overcome these limitations in order to make progress? (That is – why should people care about your contribution?)
  • 4th paragraph: Introduce your proposed approach. Describe what you do that overcomes the aforementioned limitations.
  • 5th paragraph+: Describe the benefits of your approach. Explain how you experimentally verify these benefits. Include a brief overview of the techniques and the experimental results – enough so the readers get the gist.
  • Ideally, the introduction would have a figure, a diagram, or an example (try to put it in the first page if possible) that attracts the readers’ attention and explains a key point about the problem or the approach.
• Problem Definition : Give more details on the Task Definition - Introduce the model and/or problem you are studying and define the notation you are going to use.
• Methods :
  • Explain your solution (this could take multiple sections)
  • Mention expectations – what do you expect your approach will accomplish and why.
• Experimental Evaluation (and/or) Theoretical Evaluation:
  • Often, this is where you provide the proof for the arguments made earlier in the paper. It is also an opportunity to clearly and concisely present the claimed contributions of the paper (again) and map each one to the experiments that illustrate it.
  • One good way to begin an experimental section is with (some version of) the statement: “The experiments in this section are designed to answer the following research questions:”
    • Now list 2-4 concise points that are the message of your paper, and point to where/how you prove them.
  • An experimental section will typically have three key subsections (and possible sub-subsections):
    • Experimental setting: describe your implementation and experimental setting, key parameters and how they were set (don’t do this earlier in the paper), etc.
    • Key experimental results: this is where the comparison to other approaches is done and you highlight the results of the paper.
    • Analysis: this is where you analyze you own method – ablation study or other ways. Examples are always useful in this section – both successful and sometimes failures that provide insight and allow you to discuss something interesting. Which of these is emphasized more depends on the type of the paper.
    • Note that you can use an Appendix to include fine details for each of these sub-sections, provided that the main points are in the main paper.
• Tables and Figures :
  • In most papers, tables and figures are the heart of the paper. Your message should be clear even if a reader only read your abstract and the tables and figures.
  • Imagine that this is what your readers do, and plan accordingly.
  • The information and type of presentation in the table/figure should be chosen carefully.
    • Try to avoid acronyms that are defined “somewhere” in the paper;
    • Think carefully about the information you want to give in the table and about the right way to show it.
    • Do you really need all the numbers? Do you want to show relative improvements rather than absolute numbers? (or maybe both). Is this data clearer in a bar-graph?
    • Think about statistical significance – it often strengthens your arguments if you can include it.
  • The caption, along with the table/figure need to be self-explanatory
  • Make the message of this table/figure clear in the caption (if the table/figure has no goal, don’t include it). Explain what is shown and how it supports the message.
  • Use figures also for examples – you can use it to illustrate the problem that paper addresses (in the introduction), or the difference your solution makes. And use caption to make it explicit.
• Related Work :
  • The related work section isn’t a necessary section. Sometimes it fits better at the end of the introduction, sometimes the paper focuses on analysis, and the bulk of the paper is about related work (and then a separate section isn’t needed), etc.
  • In most cases, though, you want to have a “Related Work” section, typically as the first section after the introduction, although it sometimes flows better towards the end of the paper, before the Discussion/Conclusion section.
  • In all cases, the goal of the Related Work is to show an understanding of the field and help you position the current paper relative to it.
  • Do not structure it as a “list of papers”. Rather, it should be structured to address relevant aspects of your problem and methods, that were studied in the literature.
  • Most importantly, this is an opportunity to distinguish your current work from earlier work, so for each aspect that you choose to address, use it to highlight the way in which the current work is different/better/takes a different angle. This is the real goal of this section.
• References :
  • Please use a bib file. It’s a good practice, and it makes things look nice.
  • For NLP related papers, I recommend using the ACL Anthology: https://www.aclweb.org/anthology/anthology.bib.gz
• Conclusion : summarize results. Mention the key lessons.

Project 2 Proposal Guidelines:

Please upload the proposal to Canvas; only one member per team should upload the pdf file; please name it in a way the uniquely identifies the team members.
Please submit one page structured as follows:

• Title, Team
• Your project description should include:
  • Define the problem you will address
  • Describe the resources you plan to use (eg. datasets)
  • Outline briefly the methods you current have in mind
• References (you must have some references to relevant papers on the problem and the methods you plan to use). If needed, the references can be in a second page.

Needless to say, you should strive to make the project interesting.

Project 1 Guidelines:

This project requires that you reproduce the selected paper's key results, then propose and implement a small but substantial change. The goal is for you to learn to read and understand papers at the software level.
The next steps in this project are as follows:

• Step 1 (Passed) Select one of the 4 papers proposed.
• Step 2 (Due March 8) Determine the change you want to make; by that time you have already read the paper carefully and manage to reproduce the software (even if not the results) so that you can think about what you can change. The change can be addition/deletion of key components, experiments on other datasets or any other interesting suggestion you might have. Please post your idea on Piazza. To get credit for this step, either make an original proposal, or get involved in the discussion of another proposal.
• Step 3 (Due on March 15) By that time you should have reproduced a key set of results (of your choice) in the original paper, and implemented a "change" of your choice. Note that he idea does not have to be yours, yet you have to implement it on your own. However, as in the reproduction, you are free to use any publicly available software/code-piece. The class meeting on March 15 will be devoted to presenting the results of your reproduction (success or failure) and your change. Each of the 4 papers will be presented (5 minutes; presented by 1-2 of the people working on this paper; please agree among yourselves who will present), and then each of you will get up to 4 minutes to present the highlights of your lessons (pos/neg) from this exercise, and your change.
• Submission (Due on March 15) We will ask you to submit the code you used, and a short repot (no more than 4 pages; please use this latex style) describing the problems, the results you got on an original experiment, and an empirical analysis on your "change", which could be an ablation studies (by modifying components), results on a different dataset, etc., and your conclusion.

Presentation Guidelines:

• General

  • Please include the title, authors, and publication venue (conference, journal, book chapter) of the paper you present, including publication date, on the title slide. Pointing to ArXiv isn’t sufficient since this is not a reviewed venue of publication. Add your name as the presenter.
  • Some of you may use slides or picture from other presentation (e.g., the authors) or other talks and tutorials. This is fine; but if you do it, always include credits to the source.
  • Some of the papers have demos; try to find it and use it to show examples.
  • It is good to include numbers on the slides, since it makes it easier to refer to them if needed.

• Meta-issues:

  • You do not need to present the paper in the same order the authors present it in the paper; you do not need to repeat the sales pitch of the authors as if it is true (unless it is); you do not even need to present all the information presented in the paper, if some if it is not important/essential to the overall story.
  • You many need sometimes to read beyond the paper, to understand how to place it better a context: context of prior work, context of work that followed up on it, etc.
    • Please ask me if you need additional information to help you do that better.
  • Your goal is to teach the audience what the paper is about, why it is important (or not), and place it in the context of what is/was known in the field, and in the context of the class – Learning in Few-Labels Settings.

• Presentation:

  • Examples, examples, examples, and pictures : rather than just saying things, use examples to illustrate it (in addition to text, ideally).
  • For example, try to start with presentation with an example (or two) that illustrate what the goal of the paper is, and what challenges it addresses. Only then move to your outline slide and explain the flow of your presentation.
    • Note that it is important to distinguish between that problem, and the methods/solutions. Start with the former.
  • Ideally, try to have a running example that you can use to explain the flow of ideas, the notation, and the formalism being used in the paper. It is always easier to comprehend things this way. Even when the notation and the formalism are essential, it is often best to present them alongside an example.
  • Try to avoid using vague terminology (even when the paper uses it); if it’s essential to understanding the paper, explain it. In particular, you need to realize that most people in your audience have not read the paper as carefully as you did (or at all) and you need to explain things to them.
  • Pictures and diagram are important – even when you explain the experimental setting it is useful to have a diagram that shows the relations between the settings.
  • Think about the key contribution(s) of the paper: again, context is important here. But, sometimes, the key contribution is not the contribution claimed by the authors, so you need to think about it.
  • Beyond positioning the paper and presenting the overall goal and contributions, you must go deep enough into the technical material that the paper proposes, especially if this is the key contribution of the paper.
  • Experiments: You do not necessarily need to include all the experiments, and definitely not all the numbers shown at the end of the paper.
    • I realize that it is easy to just grab things from the paper, but perhaps you can do a much better job if you choose those parts of the table(s) that actually are important to make a point (or multiple points) and teach us something. Think about it.
  • Analysis: it is always important to provide some analysis of the results; often, examples are the best way to do it – things that work and things that do not (and why).
  • Always finish your presentation with conclusion/discussion that include placing it in the context of the class.
    • What is the low-supervision paradigm addressed here? Are the authors successful from this perspective? Can it be extended one way or another? What are the things that they do not do? What are the problems in the paper? What are the shortcomings (in general, and from the perspective of the class)?

  • Details:

    • The use of animation: it is quite useful to use animation; especially if the slide has a lot of details and you want to gradually guide your audience to look at specific parts of it (e.g, if you discuss a table, you can use arrows to show people what you are currently looking at/discussing). You can also use animation to gradually introduce material presented in the slide. But don’t start with an empty slide; it isn’t pleasant to the audience and wastes time.

Discussant Guidelines:

• At the end of the talk, each of the discussant will present questions to the presenter and/or bring up a few issues relevant to the paper and the ways it addresses the supervision problem.
• The points/questions could be technical or conceptual (hopefully, you will have both).
• I designated one of the discussants as a a Pro and the second as a Con, with the hope that you will bring up different perspectives and we'll get interesting discussion going.
• By Sunday evening before the Monday presentation:
  • Please email me and Soham your at least 3 points that you would like to question/discuss.
  • You do not need to include all the details in your email, but please be prepared to present some details that support your comment.

Critical Survey Guidelines:

• Guidelines

  • A critical review will be up to one page, 11font, single-spaced; save it as a pdf (please name it: Critical Review # and your userid)
  • It should start with a clear title "Critical Review #; Paper Title; Your name.
  • Please do not spend more than a third of the page to describe what the paper does. Use the rest of the space to provide a critic of the paper:
    • What is good, what is bad; what you think is interesting; what they should have done better;
    • If possible, include suggestions for improvements and/or for additional work in this direction.
    • Any other thought you have on the paper.

• Rules

  • Do not write a critical review on the paper you are presenting or serving as a discussant for.
  • Do not write a critical review for a paper that was already presented in class. (I will make an exception for the first critical review, in case you want to write it on one of the papers presented this Monday).
  • If you plan to write a critical review on a paper that will be presented in class, make sure to send it to us before it is presented in class. This way you can contribute to the discussion of this paper.
• Here is a great critical survey written by a student (I took the name of the student out) that can be used as a reference for how to go about writing paper critics.
• Submission: On Canvas