A/B Testing
2020 to present
In this project, I will be talking about co-leading the A/B testing program at the ACLU with one Product Manager.
The program was in its nascent stages when I first started at the org and I was initially hired to find some quick win opportunities to demonstrate the value of testing on improving the ACLU’s revenue. It has since grown into a full program, used to validate product designs, optimize revenue during mission critical times, test business stakeholder strategies, and more. In my time at the ACLU, I have:
Personally designed over 50 experiments netting approximately $1mil in donation revenue lift year over year
Ideated/collaborated on more than 25 other experiments executed by a contractor that led to an additional $500k lift yoy
Process
Quarterly test generation and roadmapping
Though my team is in charge of the prioritization and roadmapping of A/B tests, we strongly believe that good ideas for tests can come from any stakeholder or teammate. I have facilitated quarterly workshops (of varying formats to experiment) with stakeholders to generate new test ideas. The goal is to generate quantity over quality. We've run workshops structured around org priorities, area/zone on the website, and user design principles.
Case study
Donation-focused navigation
My general process with any A/B test is to work with my PM on a test plan that includes hypotheses and metrics of success. The two together allow me to ideate on specific design changes I can make to pull the right levers. If the proposed test is a larger one, I workshop ideation with my internal team and relevant stakeholders.
Problem: Users are unable to find the donation type that fits their interests or motivations because the navigation hides everything except a standard donation (e.g. renewing a membership or donating monthly).
Hypothesis: By updating the main navigation to include the additional ways to donate, users will be provided clarity and relevant pathing to the donation pages, ultimately resulting in additional donations and revenue.
Exploration
Variation A: Adding a dual tier navigation that highlights our most popular donation options will clarify pathing for donors and minimize clicks.
Variation B: Using a dropdown arrow icon and updating the microcopy from “Donate” to “Give” will signal that there are several ways for donors to contribute.
+8%
Running the test for 35 days, variation B won with 83% confidence. This translated to a 4.8% increase in Average Donation Amount, and a 8% increase in Lifetime Value.
Some stats included, with standout numbers highlighted in blue:
Results
Other tests I've designed
Please see my project "Donation Form Redesign" for a largest, most comprehensive A/B test I've run to date.
Sample mocks below are presented to show breadth of ideation on five other of my most successful tests. I will happily speak about the context and results of any of the tests highlighted.
Implementation and beyond
After a test produces a winner, I am often working with both my developers and other designers to incorporate these winners, whether it be new components or new design principles, to ensure they are implemented in the most sustainable way possible. initially our program was focused on just getting our reps in andcreating test velocity, but now we're able to run 4-5 tests at a time, and so I am shifting my focus to making sure we continue to find ways to implement test winners as the real estate of the website continues to get tighter.
Future questions to explore:
Should we prioritize components based on donations or content engagement?
Can we take test winners and combine learnings into one design or component to save space and create focus for the user?
As the testing program evolves, how can we be more strategic about rolling out multi-phase tests that generate continuous learnings?
More work
A/B Testing
2020 to present
In this project, I will be talking about co-leading the A/B testing program at the ACLU with one Product Manager.
The program was in its nascent stages when I first started at the org and I was initially hired to find some quick win opportunities to demonstrate the value of testing on improving the ACLU’s revenue. It has since grown into a full program, used to validate product designs, optimize revenue during mission critical times, test business stakeholder strategies, and more. In my time at the ACLU, I have:
Personally designed over 50 experiments netting approximately $1mil in donation revenue lift year over year
Ideated/collaborated on more than 25 other experiments executed by a contractor that led to an additional $500k lift yoy
Process
Quarterly test generation and roadmapping
Though my team is in charge of the prioritization and roadmapping of A/B tests, we strongly believe that good ideas for tests can come from any stakeholder or teammate. I have facilitated quarterly workshops (of varying formats to experiment) with stakeholders to generate new test ideas. The goal is to generate quantity over quality. We've run workshops structured around org priorities, area/zone on the website, and user design principles.
Case study
Donation-focused navigation
My general process with any A/B test is to work with my PM on a test plan that includes hypotheses and metrics of success. The two together allow me to ideate on specific design changes I can make to pull the right levers. If the proposed test is a larger one, I workshop ideation with my internal team and relevant stakeholders.
Problem: Users are unable to find the donation type that fits their interests or motivations because the navigation hides everything except a standard donation (e.g. renewing a membership or donating monthly).
Hypothesis: By updating the main navigation to include the additional ways to donate, users will be provided clarity and relevant pathing to the donation pages, ultimately resulting in additional donations and revenue.
Exploration
Variation A: Adding a dual tier navigation that highlights our most popular donation options will clarify pathing for donors and minimize clicks.
Variation B: Using a dropdown arrow icon and updating the microcopy from “Donate” to “Give” will signal that there are several ways for donors to contribute.
Results
+8%
Running the test for 35 days, variation B won with 83% confidence. This translated to a 4.8% increase in Average Donation Amount, and a 8% increase in Lifetime Value.
Some stats included, with standout numbers highlighted in blue:
Other tests I've designed
Please see my project "Donation Form Redesign" for a largest, most comprehensive A/B test I've run to date.
Sample mocks below are presented to show breadth of ideation on five other of my most successful tests. I will happily speak about the context and results of any of the tests highlighted.
Implementation and beyond
After a test produces a winner, I am often working with both my developers and other designers to incorporate these winners, whether it be new components or new design principles, to ensure they are implemented in the most sustainable way possible. initially our program was focused on just getting our reps in andcreating test velocity, but now we're able to run 4-5 tests at a time, and so I am shifting my focus to making sure we continue to find ways to implement test winners as the real estate of the website continues to get tighter.
Future questions to explore:
Should we prioritize components based on donations or content engagement?
Can we take test winners and combine learnings into one design or component to save space and create focus for the user?
As the testing program evolves, how can we be more strategic about rolling out multi-phase tests that generate continuous learnings?
A/B Testing
2020 to present
Role: Design Lead (with a product manager)
In this project, I will be talking about co-leading the A/B testing program at the ACLU with one Product Manager.
The program was in its nascent stages when I first started at the org and I was initially hired to find some quick win opportunities to demonstrate the value of testing on improving the ACLU’s revenue. It has since grown into a full program, used to validate product designs, optimize revenue during mission critical times, test business stakeholder strategies, and more. In my time at the ACLU, I have:
Personally designed over 50 experiments netting approximately $1mil in donation revenue lift year over year
Ideated/collaborated on more than 25 other experiments executed by a contractor that led to an additional $500k lift yoy
Process
Quarterly test generation and roadmapping
Though my team is in charge of the prioritization and roadmapping of A/B tests, we strongly believe that good ideas for tests can come from any stakeholder or teammate. I have facilitated quarterly workshops (of varying formats to experiment) with stakeholders to generate new test ideas. The goal is to generate quantity over quality. We've run workshops structured around org priorities, area/zone on the website, and user design principles.
Case study
Donation-focused navigation
My general process with any A/B test is to work with my PM on a test plan that includes hypotheses and metrics of success. The two together allow me to ideate on specific design changes I can make to pull the right levers. If the proposed test is a larger one, I workshop ideation with my internal team and relevant stakeholders.
Problem: Users are unable to find the donation type that fits their interests or motivations because the navigation hides everything except a standard donation (e.g. renewing a membership or donating monthly).
Hypothesis: By updating the main navigation to include the additional ways to donate, users will be provided clarity and relevant pathing to the donation pages, ultimately resulting in additional donations and revenue.
Exploration
Variation A: Adding a dual tier navigation that highlights our most popular donation options will clarify pathing for donors and minimize clicks.
Variation B: Using a dropdown arrow icon and updating the microcopy from “Donate” to “Give” will signal that there are several ways for donors to contribute.
Results
+8%
Running the test for 35 days, variation B won with 83% confidence. This translated to a 4.8% increase in Average Donation Amount, and a 8% increase in Lifetime Value.
Some stats included, with standout numbers highlighted in blue:
Other tests I've designed
Please see my project "Donation Form Redesign" for my largest, most comprehensive A/B test I've run to date.
Sample mocks below are presented to show breadth of ideation on five other of my most successful tests. I will happily speak about the context and results of any of the tests highlighted.
Implementation and beyond
After a test produces a winner, I am often working with both my developers and other designers to incorporate these winners, whether it be new components or new design principles, to ensure they are implemented in the most sustainable way possible. initially our program was focused on just getting our reps in andcreating test velocity, but now we're able to run 4-5 tests at a time, and so I am shifting my focus to making sure we continue to find ways to implement test winners as the real estate of the website continues to get tighter.
Future questions to explore:
Should we prioritize components based on donations or content engagement?
Can we take test winners and combine learnings into one design or component to save space and create focus for the user?
As the testing program evolves, how can we be more strategic about rolling out multi-phase tests that generate continuous learnings?