[All skill tests](https://jobicy.com/tests.md) Skill assessment # A/B Test Statistical Inference and Experiment Analysis Skills Test This test evaluates the statistical reasoning used to design, analyze, and communicate A/B test results. It focuses on valid comparisons, uncertainty, decision rules, and practical interpretation of experimental evidence. [Start the test](#test-start) Share Assessment setup Ready ## Choose the level that matches your current experience. * 01 Basic Available * 02 Intermediate Available * 03 Advanced Available Instant result Review your score after submission 20–30 Questions per assessment 15–45 min Estimated completion time 3 levels Choose your difficulty Statistics & Experimentation [View category](https://jobicy.com/test-category/statistics-experimentation.md) Start assessment ## Choose your level and begin. Answer without outside help so the result reflects your current knowledge. You will see your score after completing the selected assessment. Reliable A/B testing requires more than comparing two conversion rates. Teams must define outcomes before launch, protect random assignment, account for uncertainty, monitor data quality, and distinguish statistically detectable effects from commercially meaningful changes. These skills support sound product, marketing, and operational decisions. Basic Intermediate Advanced This is a demo version of the test. You may attempt up to 3 questions. To take the full test and save your results, please [log in](https://jobicy.com/dashboard-page.md) or [create an account](https://jobicy.com/dashboard-page#tab2.md). This is a demo version of the test. You may attempt up to 3 questions. To take the full test and save your results, please [log in](https://jobicy.com/dashboard-page.md) or [create an account](https://jobicy.com/dashboard-page#tab2.md). This is a demo version of the test. You may attempt up to 3 questions. To take the full test and save your results, please [log in](https://jobicy.com/dashboard-page.md) or [create an account](https://jobicy.com/dashboard-page#tab2.md). Test details ## Know what to expect. Review the instructions, covered skills, example question themes, and intended audience before beginning. 01 ### Instructions and covered skills Read each scenario carefully before selecting an answer. Focus on the stated metric, population, time frame, and decision context. Do not infer details that are not provided in the question. Keep notifications off and avoid switching between tasks while completing the test. Use consistent statistical reasoning rather than relying on a single percentage change. Review each selected response for whether it addresses the experimental question directly. ### Key Areas This assessment covers the statistical judgment required to run and interpret controlled A/B tests. Candidates work with null and alternative hypotheses, p-values, confidence intervals, statistical power, minimum detectable effects, and error rates. They should understand that random assignment makes treatment groups comparable on average and that departures from expected traffic allocation can signal implementation or measurement problems. The assessment also addresses experiment design choices. This includes selecting one primary outcome, defining the unit of randomization, establishing eligibility rules, estimating sample requirements, and setting a decision rule before reviewing results. Strong performance requires recognizing threats such as repeated significance checks, metric definition changes, contaminated treatment exposure, missing conversion events, and novelty effects. Interpretation is central. Candidates must distinguish relative uplift from absolute impact, statistical evidence from business value, and aggregate results from subgroup findings. They should know when a confidence interval suggests meaningful uncertainty, why multiple comparisons can increase false positive findings, and how guardrail metrics can prevent a local improvement from causing broader harm. ### Recommended Preparation Prepare by reviewing the lifecycle of a controlled experiment: state a decision, define a primary metric, write hypotheses, select an assignment method, calculate a sample target, run a quality check, analyze the planned comparison, and document the conclusion. Practice converting between conversion counts, conversion rates, absolute percentage-point changes, and relative changes. Review the interpretation of p-values and confidence intervals without treating either as a guarantee of replication or business success. It is also useful to examine realistic experiment readouts. Identify the target population, exposure period, denominator, event definitions, treatment allocation, and guardrail outcomes before drawing conclusions. When reviewing a result, ask whether the analysis follows the original plan, whether the data collection process appears valid, and whether the observed effect is large enough to justify the operational cost or risk of rollout. 02 ### Examples of questions 1. What does a confidence interval communicate about an estimated treatment effect? 2. Why should an A/B test define its primary metric before data collection begins? 3. Which condition supports a causal interpretation of a treatment comparison? 4. What is the practical risk of checking significance repeatedly during a test? 5. How does sample size affect the precision of a conversion-rate estimate? 6. When should a team investigate a sample ratio mismatch? 7. What does statistical power represent in an experiment plan? 8. Why can a statistically detectable effect still be unsuitable for rollout? 9. How can device mix create a misleading overall experiment result? 10. What is the purpose of a holdout group in an ongoing experiment? 03 ### Who this test is best for Product analysts, growth marketers, data analysts, experimentation specialists, product managers, and optimization teams. Related assessments ## Continue with a similar skill test. Explore assessments from the same professional area to compare adjacent skills and build a broader picture of your knowledge. [Data Quality & Governance 3 levelsData Lineage Mapping and Impact Analysis Skills TestAssess the ability to document data movement, trace transformations, and evaluate downstream consequences of change. This test focuses on practical lineage records that… Open assessment →](https://jobicy.com/tests/data-lineage-mapping-and-impact-analysis-skills-test.md) [Excel & Google Sheets 3 levelsExcel and Google Sheets Lookup Formula Design Skills TestThis test evaluates the ability to retrieve, match, and validate data with lookup formulas in Excel and Google Sheets. It focuses on selecting… Open assessment →](https://jobicy.com/tests/excel-and-google-sheets-lookup-formula-design-skills-test.md) [SQL & Databases 3 levelsPostgreSQL Index Design and Query Plan Analysis Skills TestEvaluate how well you design PostgreSQL indexes and interpret query execution plans. The test focuses on matching index structures to query predicates, joins,… Open assessment →](https://jobicy.com/tests/postgresql-index-design-and-query-plan-analysis-skills-test.md) [Data Interpretation & Visualization 3 levelsTableau Dashboard Design and Visual Encoding Skills TestAssess the ability to design Tableau dashboards that communicate data clearly and support sound decisions. The test focuses on chart selection, visual encodings,… Open assessment →](https://jobicy.com/tests/tableau-dashboard-design-and-visual-encoding-skills-test.md) [Business Metrics & KPI Analysis 3 levelsSaaS Cohort Retention and Churn Analysis Skills TestThis test evaluates the ability to build, interpret, and act on SaaS customer retention cohorts. It focuses on consistent cohort definitions, retention calculations,… Open assessment →](https://jobicy.com/tests/saas-cohort-retention-and-churn-analysis-skills-test.md) [Business Metrics & KPI Analysis 3 levelsRole-Specific KPI Identification Skills TestThis test evaluates your ability to select the most relevant Key Performance Indicators for various digital roles. Master this skill to ensure your… Open assessment →](https://jobicy.com/tests/role-specific-kpi-identification-skills-test.md) [SQL & Databases 3 levelsSQL JOINs and Multi-Table Queries Skills TestMaster the art of combining data from multiple tables using SQL JOINs. This test evaluates your ability to write efficient queries using various… Open assessment →](https://jobicy.com/tests/sql-joins-and-multi-table-queries-skills-test.md) [Data Interpretation & Visualization 3 levelsData Storytelling Techniques Skills AssessmentThis assessment evaluates your ability to craft compelling narratives using data. It covers key techniques for turning raw numbers into actionable insights. Open assessment →](https://jobicy.com/tests/data-storytelling-techniques-skills-assessment.md) [Excel & Google Sheets 3 levelsPivot Tables and Data Summarization Skills TestThis test assesses your proficiency with pivot tables in Excel and Google Sheets, focusing on data summarization, grouping, and analysis techniques. Ideal for… Open assessment →](https://jobicy.com/tests/pivot-tables-and-data-summarization-skills-test.md) ## Share the assessment or try another skill. Send this test to a colleague or friend, or return to the assessment library to explore another professional area. Share this test [Browse all tests](https://jobicy.com/tests.md)