Extended Seminar – Systems and Machine Learning

The sem­i­nar page can be found here in TUCaN. Additional material will be provided via Moodle.

Who, when and where?

The sem­i­nar will be held as block sem­i­nar.

The days for talks and deadlines will be decided in the seminar.
The kick-off meet­ing will be on October 18, 1:00-2:00pm (Room: S2|02 A213).

The sem­i­nar will be joint­ly held by Profs. Carsten Bin­nig, Kris­tian Ker­st­ing, Andreas Koch, and Mira Mezini.

Pre­req­ui­sites

It is not nec­es­sary to have prior knowl­edge in ar­ti­fi­cial in­tel­li­gence, but prior knowl­edge in software/hardware systems and ma­chine learn­ing is helpful.​ Partic­i­pa­tion is lim­it­ed to 20 students.​

For fur­ther ques­tions feel free to send an email to dm@​cs.​tu-darmstadt.​de. No prior reg­is­tra­tion is need­ed, how­ev­er, please still send us an email so that we are able to es­ti­mate be­fore­hand the num­ber of par­tic­i­pants, and have your E-mail ad­dress for pos­si­ble announcements.​ Also make sure that you are reg­is­tered in TUCaN.

Con­tent

This seminar serves the purpose of discussing new research papers in the intersection of hardware/software-systems and machine learning. The seminar aims to elicit new connections amongst these fields and discusses important topics regarding systems questions machine learning including topics such as hardware accelerators for ML, distributed scalable ML systems, novel programming paradigms for ML, Automated ML approaches, as well as using ML for systems.

The top­ics will be as­signed based on an on-line bid­ding pro­cess, which will be opened after the kick-off.​ The final as­sign­ment will be made a week later.

Ex­tend­ed Sem­i­nar

What is “Ex­tend­ed” about this sem­i­nar? Stu­dents are not only ex­pect­ed to give a short talk, but also to pre­pare a small write-up.​ The write-up will be pre­pared in groups, each group will cover one theme, con­sist­ing of four topics.​ The final write-up must be con­cise and short, and should give a short overview of the theme (not nec­es­sar­i­ly lim­it­ed to the stud­ied pa­pers).

In ad­di­tion, we will also do a peer re­view­ing pro­cess, as it is usu­al­ly done at sci­en­tif­ic conferences.​ This means that you also have to read (some) of the other write-ups and pro­vide feed­back by fill­ing out a re­view form.

Be­cause they are more work for stu­dents, stu­dents re­ceive 4 CPs for Ex­tend­ed Sem­i­nars (in­stead of 3 CPs for reg­u­lar sem­i­nars).

Talks

Al­though each topic is typ­i­cal­ly as­so­ci­at­ed with a sin­gle paper, the point of the talk is not to ex­act­ly re­pro­duce the en­tire con­tents of the paper, but to com­mu­ni­cate the key ideas of the meth­ods that are in­tro­duced in the paper.​ Thus, the con­tent of the talk should ex­ceed the scope of the paper, and demon­strate that a thor­ough un­der­stand­ing of the ma­te­ri­al was achieved.​ See also our gen­er­al ad­vices on giv­ing talks.

Stu­dents are ex­pect­ed to give a 20 (!) minute talk on the ma­te­ri­al they are as­signed, fol­lowed by 10 min­utes of questions.​ Note that the com­pa­ra­bly short pe­ri­od of time forces you to get the most im­por­tant points of your topic across.​ You are not ex­pect­ed to pre­sent ev­ery­thing.

The talks are ex­pect­ed to be ac­com­pa­nied by slides.​ In case you do not own a lap­top, please send us the slides in ad­vance, so that we can pre­pare and test the slides.​ The talk and the slides should be in En­glish.

The talks will be presented in a block on 10 and 11 January 2019.

Write-Up

The talks are or­ga­nized in top­i­cal groups.​ Each group must pre­pare one short write-up of their work.

Con­tent: The pa­pers are re­lat­ed to each other.​ Your task is to use these pa­pers to cre­ate a mi­ni-sur­vey that com­bines the re­sults of all pa­pers, and pos­si­bly other papers.​ The con­tri­bu­tion of each in­di­vid­u­al paper can be lim­it­ed to the most im­por­tant points that are con­tribut­ed by this paper to the topic.​ There must be a clear “red thread” with­in each sur­vey, a con­cate­na­tion of in­di­vid­u­al paper sum­maries is not enough.​ A pos­si­ble out­line can con­sist of an in­tro­duc­tion to set the stage and out­line the cross-cut­ting themes of all pa­pers, mul­ti­ple sec­tions on in­di­vid­u­al con­tri­bu­tions w.​r.​t.​ cross-cut­ting themes and com­par­i­son of dif­fer­ent ap­proach­es, a joined re­lat­ed work sec­tion, and a sum­ma­ry and out­look.

For­mat: The for­mat for the write-up is pre­de­fined, and fol­lows con­ven­tions that are typ­i­cal­ly used for pub­li­ca­tions in com­put­er science.​ In par­tic­u­lar, we re­quire each paper to be for­mat­ted ac­cord­ing to the Tem­plate for Proceedings in ACM Conferences (2-column layout). Each paper should have no more than 6 pages in this for­mat (the bib­li­og­ra­phy is not count­ed, and can be as long as nec­es­sary). The for­mat must not be changed in order to gen­er­ate more space.​ Each paper also must, of course, have a title, au­thors, and an abstract.​ The tem­plates are avail­able in Word and LaTeX, but we strong­ly rec­om­mend that you try to use LaTeX.​ Environ­ments such as MiK­TeX and TeXs­tu­dio make local La­TeX-edit­ing quite easy, and web-sites like Over­leaf offer col­lab­o­ra­tive work­ing en­vi­ron­ments for LaTeX.

Dead­line: The write-ups are due 29 January, 2018.

Re­view­ing

Re­view­s are required for all three other writeups.​ A review­ing form will be pro­vid­ed by then.​ The dead­line of the stu­dents’ re­views will be 19 February, 2018.

Grad­ing

The slides, the pre­sen­ta­tion, the an­swers given to ques­tions in your talk will in­flu­ence the over­all grade, as will the write-up and the reviews.​ Further­more, it is ex­pect­ed that stu­dents ac­tive­ly par­tic­i­pate in the dis­cus­sions, and this will also be part of the final grade.

To achieve a grade in the 1.​x range, the talk and write-up needs to ex­ceed the recita­tion of the given ma­te­ri­al and in­clude own ideas, own ex­pe­ri­ence or even examples/demos.​ An exact recita­tion of the pa­pers will lead to a grade in the 2.​x range.​ A weak pre­sen­ta­tion and lack of en­gage­ment in the dis­cus­sions may lead to a grade in the 3.​x range, or worse.​ For the write-ups it is im­por­tant that they pro­vide a co­her­ent view (like a sur­vey paper), and do not sim­ply con­sist of a con­cate­na­tion of four paper sum­maries.

Top­ics and Sched­ule

All pa­pers should be avail­able on the in­ter­net or in the ULB.​ Note that Springer link often only works on cam­pus net­works (some­times not even via VPN). If you can­not find a paper, con­tact us.

Machine Learning and Data Management

Machine Learning to enhance Database Systems (Binnig)
  • (A1) Tim Kraska, Alex Beutel, Ed H. Chi, Jeffrey Dean, Neoklis Polyzotis: The Case for Learned Index Structures. SIGMOD Conference 2018.
  • (A2) Ma, Lin, et al. Query-based Workload Forecasting for Self-Driving Database Management Systems. SIGMOD Conference 2018.
  • (A3) Krishnan, S., Yang, Z., Goldberg, K., Hellerstein, J., & Stoica, I. Learning to optimize join queries with deep reinforcement learning. arXiv 2018
  • (A4) Kipf, A., Kipf, T., Radke, B., Leis, V., Boncz, P., & Kemper, A. Learned Cardinalities: Estimating Correlated Joins with Deep Learning. arXiv 2018.
  • (A5) Li, T., Xu, Z., Tang, J., & Wang, Y. (2018). Model-free control for distributed stream data processing using deep reinforcement learning. PVLDB 2018
Machine Learning for Knowledge Base Construct (Kersting)
  • (B1) Ismail Ilkan Ceylan, Adnan Darwiche, Guy Van den Broeck: Open-World Probabilistic Databases. KR 2016: 339-348.
  • (B2) Benny Kimelfeld, Christopher Ré: A Relational Framework for Classifier Engineering. SIGMOD Record 47(1): 6-13 (2018).
  • (B3) Ce Zhang, Christopher Ré, Michael J. Cafarella, Jaeho Shin, Feiran Wang, Sen Wu: DeepDive: declarative knowledge base construction. Commun. ACM 60(5): 93-102 (2017).
  • (B4) Parisa Kordjamshidi, Dan Roth, Kristian Kersting: Systems AI: A Declarative Learning Based Programming Perspective. IJCAI 2018: 5464-5471.

Machine Learning Systems

Distributed Machine Learning (Binnig)
  • (A6) Mu Li, David G. Andersen, Jun Woo Park, Alexander J. Smola, Amr Ahmed, Vanja Josifovski, James Long, Eugene J. Shekita, Bor-Yiing Su: Scaling Distributed Machine Learning with the Parameter Server. OSDI 2014
  • (A7) Philipp Moritz, Robert Nishihara, Stephanie Wang, Alexey Tumanov, Richard Liaw, Eric Liang, William Paul, Michael I. Jordan, Ion Stoica: Ray: A Distributed Framework for Emerging AI Applications. Arxiv 2017
  • (A8) Jiawei Jiang, Fangcheng Fu, Tong Yang, Bin Cui: SketchML: Accelerating Distributed Machine Learning with Data Sketches. SIGMOD Conference 2018
  • (A9) Hantian Zhang, Jerry Li, Kaan Kara, Dan Alistarh, Ji Liu, Ce Zhang:
    ZipML: Training Linear Models with End-to-End Low Precision, and a Little Bit of Deep Learning. ICML 2017
  • (A10) Anthony Thomas, Arun Kumar: A Comparative Evaluation of Systems for Scalable Linear Algebra-based Analytics. PVLDB 2018
  • (A11) Tian Li, Jie Zhong, Ji Liu, Wentao Wu, Ce Zhang: Ease.ml: Towards Multi-tenant Resource Sharing for Machine Learning Workloads. PVLDB 2018
Automating Machine Learning (Kersting)
  • (B5) Alexander J. Ratner, Christopher De Sa, Sen Wu, Daniel Selsam, Christopher Ré: Data Programming: Creating Large Training Sets, Quickly. NIPS 2016: 3567-3575.
  • (B6) Matthias Feurer, Aaron Klein, Katharina Eggensperger, Jost Tobias Springenberg, Manuel Blum, Frank Hutter: Efficient and Robust Automated Machine Learning. NIPS 2015: 2962-2970.
  • (B7) Yutian Chen, Matthew W. Hoffman, Sergio Gomez Colmenarejo, Misha Denil, Timothy P. Lillicrap, Matthew Botvinick, Nando de Freitas:
    Learning to Learn without Gradient Descent by Gradient Descent. ICML 2017: 748-756.
  • (B8) Antonio Vergari, Alejandro Molina, Robert Peharz, Zoubin Ghahramani, Kristian Kersting, Isabel Valera: Automatic Bayesian Density Analysis. CoRR abs/1807.09306 (2018).
  • (B9) James Robert Lloyd, David K. Duvenaud, Roger B. Grosse, Joshua B. Tenenbaum, Zoubin Ghahramani: Automatic Construction and Natural-Language Description of Nonparametric Regression Models. AAAI 2014: 1242-1250.

Machine Learning and Software Engineering

Programming Abstractions for Machine Learning (Mezini)
Machine Learning for Software Engineering (Mezini)

Hardware for Machine Learning  (Koch)