Identifying Research Problems Michael D. Ernst IMDEA Software

Identifying Research Problems Michael D. Ernst IMDEA Software Institute and University of Washington Workshop at UPM

Introductions I do research in programming langages, software engineering, security, and related fields Professor at the U. of Washington on sabbatical at IMDEA Software Institute previously Associate Professor at MIT & researcher at Microsoft Research maintain a webpage with advice: https://homes.cs.washington.edu/ mernst/advice/ Who are you? (name, year, topic, research experience, why you are here, what you hope to learn)

The hardest part of research: asking questions If you are smart enough to get into graduate school, you are probably good at answering questions That’s what college trained you to learn Graduate school teaches you how to ask questions Once you ask the right question, the answer is often obvious The question you ask implies a framing of the problem

The characteristics of great research Recall a good research paper/talk/result What was remarkable about it?

The characteristics of great research Great research needs: idea today’s focus execution presentation We will

Well-presented research convinces you: 1. The problem is interesting about the problem effect on human welfare, corporate profits, defense evidence: other people have worked on it 2. The problem is hard other smart people have tried it obvious approaches do not work problem persists over time 3. The authors have solved the problem technical details evaluation, such as proofs or experiments

Sources of research ideas Always be working on something (even if it won’t be your thesis topic) Think about problems, difficulties, or irritations You might get sidetracked onto something more important Learn about other research: attend talks, read papers, take classes future work they identified connections they did not make assumptions other criticisms Your advisor OK for starter projects, master's projects not OK for PhD thesis topic

Think outside the box: A good research idea often “cheats” If you take the same approach as other people, you are likely to be no more successful than they are Cut the Gordian knot: take a different approach Example: Artificial intelligence vs. brute force (chess, speech, etc.)

Bad ideas are essential Most ideas are bad ideas If you have lots of ideas, some will be good It’s better to have bad ideas than no ideas I can teach people how to filter It’s harder to teach courage and imagination Filtering is initially your advisor’s role You will get better with time, and spend less time evaluating and discarding bad ideas Have faith in your own judgment, too Your advisor is not infallible. Examples: Daikon, IGJ Generate lots of ideas before filtering Breadth-first, not depth-first, approach to the problem

Heilmeier’s Catechism (questions to be asked about any research proposal) 1. What are you trying to do? Articulate your objectives using absolutely no jargon. 2. How is it done today, and what are the limits of current practice? 3. What's new in your approach and why do you think it will be successful? 4. Who cares? 5. If you're successful, what difference will it make? 6. What are the risks and the payoffs? 7. How much will it cost? 8. How long will it take? 9. What are the midterm and final "exams" to check for success?

Brainstorming strategies

Automation Find a task that is currently done manually, or that is time-consuming or difficult Thoughtfully examine the manual process What reasoning does the person do? Can you automate that process?

Contrarianism Turn the conventional wisdom on its head Do the opposite of what everyone else expects You must do this thoughtfully: you have to justify why, and why it matters Example: sequentializing parallel programs Solves the same problem, and is better in some ways

Notice complementary approaches Given two approaches to a problem: wherever one is used, try the other one Example: static and dynamic analysis Static: sound, conservative Dynamic: precise, does not generalized dynamic invariant detection Example: semantic vs. syntactic analysis

Combine existing techniques Apply one after the other Create a hybrid analysis

Cross-fertilization Take an approach from one domain, and apply it to another Examples: Machine learning, statistics, biology,

Fill in gaps Review your previous research Are there themes or commonalities? Find ways to fill in the gaps This is particularly useful for creating a PhD thesis out of a sequence of successful projects.

Goal vs. technique A research project requires: A goal or problem A technical approach Advantages of starting with a goal More likely to find a technical approach More likely to be novel and relevant Advantages of starting with a technique Quick to complete: you probably already have it implemented Fits well with your other work

What is your secret weapon? Use your unique set of skills and experiences Why are you the right person to work on this? You have taken a particular combination of classes Example: Steve Jobs and calligraphy class You have a friend with a tool/technique/knowledge You thought of it first (new approach) You are willing to work harder

Passion for your work Your intellectual curiousity about a topic is necessary but not sufficient. Necessary: Your love for the topic will affect your motivation (and possibly ability) It helps if the topic was your idea Not sufficient: It has to matter to other people as well

Generalizing How does this generalize? What are the broader implications?

Brainstorming exercise Think about problems that you encounter in day-to-day programming.