From today, all content for the ICLR2020 Virtual Conference is available in open-access for anyone across the world to learn from. A public archive of the virtual conference site is now available for everyone to explore the 2020 conference proceedings, and to get a sense of the virtual conference portal and its flow. The registered participants site remains available.
Organising the 8th international conference on learning representations (ICLR 2020) was highly challenging, but ultimately, highly rewarding for our organising committees. Our initial work for hosting the conference in Ethiopia was far along, but then COVID-19 pandemic gave us the opportunity to create a new way of hosting our conferences. We look back now on the virtual conference week satisfied that we were able to show that a virtual conference is a viable way for us to share our science and build an international machine learning community, and that many meaningful avenues for connection are already possible.
We thank everyone for their support, patience and kindness in this time of change and experimentation. We thank the authors, organisers of workshops and socials, our volunteers, and everyone who registered. Together you made ICLR2020 a reality and a success.
We know that there is much more to do to create a better virtual conference experience for everyone. We hope this first virtual conference is one we can collectively build upon. Towards that aim, this post summarises some of our key reflections on the design and experience of the ICLR2020 Virtual Conference.
🎨 Design Principles
We used two key principles — to be Inclusive and Interactive — in our approach to the virtual conference. To these principles we returned whenever we found ourselves needing to make a difficult decision or tradeoff between competing approaches. What did these principles mean for us?
Inclusive
- Asynchronous-first. The conference had to be time-zone agnostic, so that everyone could participate within healthy working hours, and when they could. This meant that the conference had to be asynchronous-first, so that no one needed to be awake at odd hours, everyone could create their own flow of activities, catch up on anything missed, or skip ahead as needed.
- Extended live sessions. Live components had to be spread as evenly as possible across the world. For this reason we had 5 poster sessions each day at time zones across the world, and every paper was presented twice (in 2 separate sessions).
- Globally accessible. All parts of the conference had to be accessible everywhere. This meant we couldn’t use tools and services that only worked in certain countries, or that did not have easy ways to overcome any limitations.
- Low cost. The cost of the participation should not be a barrier. We created several opportunities for free registration, and lowered the fee to as low as we could, while still allowing income to pay for the services that were needed.
Interactive
- Chat everywhere. We needed to support as many forms of interaction as we could. Combined with the principle of inclusivity, this meant that every component of the conference had a chat channel associated with it to allow asynchronous, direct, and open conversation.
- Face-to-face. Poster sessions happened over video calls that took place at two times for every paper. We also hosted an extended live Q&A with every keynote speaker that allowed many more people than usual to directly ask questions of our speakers.
- Social. We asked for social meetups to be hosted that would allow the community to meet around shared interests and themes.
- Self-organising. We wanted there to be opportunities for direct and impromptu ways of interacting. We experimented with on the fly mentoring sessions, ad-hoc meetings, and having repeated socials.
🎒Lessons and Recommendations
We brought the entire virtual conference together within a 6 week period, so many of the issues we identified could be addressed with more deliberation and time. Overall, we believe we achieved all our goals with this approach to a virtual machine learning conference. We share some of our reflections and sites for improvement here.
Organisation and Communication
- Early communication. The virtual format is new, and requires a lot of scheduling to be done upfront. Being able to communicate early and in small chunks would have helped us smooth out confusion, build up an understanding of the virtual format, and help everyone be more prepared for this new format.
- Fees and scholarships. The virtual format opens up new avenues for inclusion, meaning that the programmes we had to support participation needed to grow proportionately. They needed to be clear, easy to find, and allow time for applications. We did support many free registrations through our volunteers, with additional registration support through our community groups. Having a clear place for this, and then multiple channels for outreach will help these programmes better reach those who need them.
- Volunteers. We had 500 volunteers who were a highlight of the conference for us. Our volunteers were energetic, spread across the world, and committed to improving the virtual experience for everyone. Our volunteers received free registrations in exchange for supporting the conference in several roles. Volunteers had early access to the conference site and helped us stress-test all aspects of the system. They were an on-call help team throughout the conference to help anyone who had technical questions or difficulties. A volunteer programme is one we highly recommend.
- Attendance and Engagement. We had a higher number of attendees than we expected to get. The physical conference in 2019 had 2700 attendees, and the virtual in 2020 had 5600 attendees. It is likely that, without the typical funding and visa restrictions for physical travel, more individuals were able to participate. This number remained highly manageable, because interaction was asynchronous across time zones. At any given time, maybe 200–300 people were engaged in an event. We also suspect engagement was skewed, with some participating in many events and others rarely present. We suspect some could not engage fully, because they did not take time off of work or had difficulties engaging from home rather than a dedicated physical space. A culture shift in engagement for virtual events is needed for better engagement.
Conference Components
- Pre-recorded videos. All the talks were pre-recorded and made available at the start of the conference. The ability to watch the invited talks and discuss questions (and upvote questions) before the Q&A made for a highly engaging and informative Q&A session. The availability of a short 5 minute video for each paper was also very useful for getting the main message of the work. The quality of all the videos was surprisingly good, and the system for recording was simple to use.
- Poster Assignments. Assignments worked overall and we struck a good balance between the needs of authors and the availability of events across the world. More time would have prevented missing and incorrect time requests from authors, and allowed us to iterate on the scheduling with the authors.
- Poster Interactivity. Posters sessions done over video need to be approached differently than what we are used to at a physical conference. There is a missing energy that is usually shared across presenters and attendees. Presenters are disconnected from their audiences and didn’t know how many people were engaging with their posters/videos. There is a reluctance from participants to join calls and be the only person, and a sense of pressure to have either read the paper or have insightful or ‘intelligent’ questions. This is where most iteration will be needed in future, by finding a way for authors to see the watch counts of their videos, the number of people on a video-call already, and in other creative virtual environments. But this also requires a change in the conception of how poster sessions work from everyone, and is something that will evolve as we all get used to these new forms of engagement.
- Speaker Q&A. The speaker session worked out as hoped: they allowed us to have more meaningful engagement with our speakers, reduced the barrier to asking questions by having multiple routes for questions to be asked, and allowed many more people than usual to ask questions. Even better, many of our speakers joined their chat session after their session and continued to answer questions over chat (for over an hour more in one case). Times for the live sessions were chosen for the convenience of the speaker, but the videos of these sessions were available immediately afterwards, so anyone who missed it live could watch it whenever they could.
- Workshops. Workshops were more difficult to design. They required a significantly larger amount of coordination between technical teams and chairs. We also required much more communication with individual workshop organisers who needed to provide details of their workshop timings, speakers, help develop an understanding of technical toolset and flow, website needs, etc. In future, this requires a slightly larger team of chairs, and other technical tools to allow components of the main conference website to be customised by workshop organisers for their own use.
Interactivity and Socials
- Geographic-spread of socials. We had a call for virtual meetups to be hosted and that would be supported by the conference, ultimately hosting 29 socials throughout the week. The set of socials we had were not as spread across global locations as we would have liked, and more targeted outreach would have allowed us to support social events spread across the world. The coverage of topics was also incomplete; again, with more time, we would likely have made calls for particular topics for socials, to facilitate interaction between attendees.
- More interactivity. We found that groups self-organised over chat for meetups on topics. A community-driven mentorship scheme also emerged where more experienced researchers would answer questions over video chat with younger researchers about research questions and career paths. Future improvements could be to pre-arrange video meet-and-greet sessions (either random or matched by interest, like the neuromatch approach) and to make such mentorship part of the conference programme, and spread across the conference week. We could also create standing virtual coffee sessions in future, and explore other ways for interactivity. Another component we experimented with was ICLRTown, which is a virtual environment that allowed participants to interact with each other in a virtual environment, which had a small user base, but received a positive reaction.
Technical Systems and Support
- Chat, Video, Pre-recorded video. We chose tools that worked everywhere that allowed us to control data and use, work everywhere across the world, were open-source projects if possible, and simple to integrate into our systems. We had no problems in using SlidesLive, Rocket.Chat, and Zoom for these needs and would use them all again.
- Single sign-on. Reducing the technical system to a single sign-on would have made the experience simpler and more convenient for everyone. In addition, it would have allowed us to implement more features for safety and privacy.
- Web development. We identified early on that a portal bringing together all the different tools we needed was not available. We explored some of the available platforms, but decided that we could best serve the culture and needs of our conference by creating this platform ourselves. This meant that a burden of the technical development load fell to members of our team. Having web developers that can be part of the central team and that take on this load will be needed in future.
- On-call support. Our volunteers were an on-call support throughout the day and across the world to support participants queries using the #helpdesk channel, which worked extremely well. There were 2 incidents of ‘zoom bombing’ during the conference because authors shared their links publically. Addressing this requires even more on-call support, to be able to create new video links and update this across the site. Other ways of preventing this, through shared responsibility across the community and other technical tools will need to be developed in future.
- Machine Learning in the design of the conference. Where it made sense, we wanted to use machine learning in the design of the conference programme and experience itself. We used a latent variable model for review score calibration, a vision model to extract thumbnails from each paper to be used on the web, natural language tools to visualise related papers, and recommender systems to create sets of balanced recommendations for papers and participants. The virtual conference creates a new environment within which we can try out and explore many of the machine learning approaches that the conference itself showcases.
🏁 Getting Ready for ICLR 2021
- Congratulations 🎉 Our best wishes and congratulations to the ICLR 2021 programme chairs: Katja Hofmann (senior PC), Alice Oh, Ivan Titov, and Naila Murray.
- Be an ICLR2021 Reviewer: If you have not reviewed for ICLR before and want to be considered as a reviewer or an area chair for the next round, please complete this short form: https://surveymonkey.co.uk/r/QFBT67V
- Next Deadline: Next deadline will be the beginning of October 2020, with the conference in the first week of May 2021. To be or not to be virtual, to be decided.
Signing off, Your ICLR2020 Organising Committees
Sasha Rush and Shakir Mohamed (General Chair and Senior Programme Chair); Dawn Song, Kyunghyun Cho, Martha White (Programme Chairs); Asja Fischer and Gabriel Synnaeve (Workshop Chairs); Adam White (Social Chair), Hendrik Strobelt (Virtual Chair); Anima Anandkumar and Kevin Swersky (Diversity Chairs); Timnit Gebru and Esube Bekele (Logistics Chairs); Andrea Brown and Lee Campbell (ICLR Secretariat), and the ICLR Board.
We’ll now be taking a much needed break from conferencing. We thank you for your support during these uncertain times. We wrote this, collecting our final notes for the next committees, and listening to Started from the Bottom by Drake 🎧.
