Remote Sr. Data Scientist – Psychometrics @ Udemy

You’re an analytical problem-solver ready to put your skills toward purposeful work that has a global impact. You want to lead the way in innovation, exploring the latest technologies and finding new solutions. You thrive in a collaborative environment and are eager to work with and learn alongside the best in Product, Design, and Engineering.

About this role:

In the Data organization at Udemy, we’re passionate about transforming the future of education using data!  Psychometrics plays an important role in Udemy’s growing skills validation strategy. We’re looking for fun, collaborative, self-motivated psychometricians with an insatiable sense of curiosity and a knack for asking the right questions. Our ideal candidate is an analytically creative thinker who is unafraid of ambiguity, has a track record of tackling difficult technical challenges, and has a pragmatic and relentless focus on driving meaningful business impact. We believe in building diverse teams, collaborative teams in which each person brings unique skills, knowledge, and experience to the table. If this opportunity sounds exciting to you, then read on! As a psychometrician at Udemy, you will be partnering with product managers, designers, and engineers to build algorithmic and model-driven solutions to power skill assessments. This role will require a combination of applied measurement experience, data-driven decision making, and the data science and coding chops to dive headfirst into development on production systems. A successful candidate will have knowledge of best industry practices for growing and maintaining a large set of assessment products, while being courageously experimental in finding new solutions to ongoing issues in test development and maintenance.

What you’ll be doing:

• Collaborate with other psychometricians, data scientists, PMs, designers, and engineers to maintain and grow our skill assessment platform
• Effectively communicate and present to both technical and non-technical audiences
• Critically evaluate and implement the state-of-the-art research literature in educational assessment
• Develop production-grade data pipelines and continuously improve the quality and accessibility of data
• Develop psychometric models for skill assessment through all phases, including prototyping, productionization, and experimental evaluation
• Define metrics and build dashboards to track the adoption and health of skill assessments
• Collaborate with content developers and instructional designers to develop and refine item writing and review processes

What you’ll have:

• 3+ years of hands-on industry experience in a psychometric or data science-oriented role
• Domain experience in educational assessment and psychometrics (e.g., Classical Test Theory (CTT), Item Response Theory (IRT), Computerized Adaptive Testing (CAT), equating and scaling)
• Experience creating or applying novel measurement models (Diagnostic Classification, Knowledge Tracing, MIRT) and their associated estimation algorithms
• Strong SQL ability and proficiency with 1+ programming languages (e.g., R or Python)
• End-to-end experience organizing and analyzing large data sets for operational purposes
• Ability to communicate effectively with non-technical stakeholders
• Strong knowledge of applied statistics and experimental design
• Ability to work autonomously, relentlessly prioritize, and collaborate with team members across time zones

We understand that not everyone will match each of the above qualifications. However, we also realize that everyone has unique experiences that can add value to our company. Even if you think your background might not perfectly align, we’d love to hear from you!

We Are More Excited About You Because You Have:

• 5+ years of relevant industry experience
• Experience with machine learning and causal inference
• Familiarity with standard software engineering practices (e.g., Scrum)
• Experience using big data technologies (e.g., Hadoop, Spark)

Life at Udemy:

We aspire to be as vibrant and dynamic as the communities we serve, as inquisitive as those who use our platform, and as revolutionary as the future we strive to open for everyone. Here are some of the things we love about life at Udemy: We’re invested in creating an inclusive environment that welcomes a diverse range of backgrounds and experiences. From creating employee resource groups, ensuring we’re a Fair Pay Workplace, and building a flexible work culture, our belonging, equity, diversity, and inclusion (BEDI) initiatives always put our people first. We want you to be able to bring your authentic self to work because when we all do, we’re better for it. Learning is what we do – inside and out. Our Learning & Development team is second to none, helping ensure your journey is one of continuous progression. You’ll also have unlimited access to Udemy courses, monthly UDays (meeting-free professional development days), and a generous annual professional development stipend. Our reason to exist is to revolutionize learning – that calls for taking risks and learning from failures. Whether it’s our hackathons (a company-wide effort to envision new possibilities for our product) or sharing our prototypes, we see experimentation as a crucial step on the path to success. We’re committed to creating world-class employee experiences and are proud of the recognition of this by Great Place to Work. Of course, the best thing about being part of Udemy is knowing your work makes a difference for people and organizations around the world. You’ve got the skills; why not use them to help others develop theirs? At Udemy, we value diversity and inclusion and consider qualified applicants without regard to race, color, religion, sex, national origin, ancestry, age, genetic information, sexual orientation, gender identity, marital or family status, veteran status, medical condition, or disability. We will consider for employment qualified applicants with arrest and conviction records.