TrackIt / AWS · 2024 · Product Manager

WAFR Automation Tool

A SaaS platform that automates AWS Well-Architected Framework Reviews, replacing a manual consulting process with a repeatable, scalable product available as open source, managed SaaS, or on-prem deployment.

Product StrategyUser ResearchAWSSaaSPricing & Packaging

Overview

ClientTrackIt / AWS

Date2024

RoleProduct Manager

Scope of workProduct Strategy, User Research, AWS, SaaS, Pricing & Packaging

The context.

I positioned the WAFR Automation Tool as a way to productize Well-Architected Reviews, making them faster, more consistent, and scalable. As Product Manager, I owned the product end-to-end, from discovery and strategy to delivery, pricing, and positioning.

From a business perspective, the product addressed two core needs: customers needed quicker, more actionable reviews across multiple AWS accounts, and TrackIt needed a scalable alternative to expert-led, manual assessments.

Delivery models

Open Source: Free, limited functionality. No pillar mapping, no AI-driven insights. Entry point for discovery and community adoption.

As-a-Service: Paid per assessment. Includes full Well-Architected pillar mapping, automated findings, and AI-driven recommendations.

On-Prem: Monthly subscription deployed inside the customer's own AWS environment. Designed for security-constrained accounts and enterprise requirements.

Automated mapping of AWS resources to the five Well-Architected Framework pillars.

Discovery

The discovery.

I conducted discovery with cloud engineers, AWS architects, and consulting teams already performing Well-Architected Reviews. Discovery focused on real operational workflows, not theoretical requirements.

Key pain points

Heavy manual data collectionEvery review started from scratch, requiring engineers to manually query and document AWS resource configurations across accounts.

High dependency on individual expertiseReview quality varied significantly depending on which consultant ran it, with no shared baseline or repeatable methodology.

Inconsistent findingsThe same environment reviewed twice could produce different outputs, undermining customer trust in the process.

Long time-to-valueReviews took weeks to complete, delaying the recommendations customers needed to act on.

Limited scalabilityAs AWS environments grew in complexity, the manual approach could not keep pace with the scope or speed of change.

Approach

What I built.

I designed the product as an automation layer on top of AWS accounts, working with AWS-native tools including CloudSploit and CloudCustodian to power the analysis engine. My role was to translate a conceptual framework into a system that could run consistently at scale.

Key product decisions

Separate data collection, analysis, and presentation layers for independent iteration

A rules-based engine aligned directly with Well-Architected pillars

Structured, actionable outputs instead of raw data dumps

Safe execution model designed for customer AWS environments

Roadmap structure

1.Core account analysis and data extraction

2.Mapping AWS resources to Well-Architected pillars

3.Actionable findings and recommendations generation

4.Flexible deployment and pricing models

Packaging

Pricing & packaging.

I defined three product offerings, each reflecting a different customer maturity level, security constraint, and value delivered. Pricing was designed to grow with the customer rather than gate them out early.

Open SourceFree, limited functionality. No pillar mapping, no AI-driven insights. Entry point for discovery and community adoption.

As-a-ServicePaid per assessment. Includes full Well-Architected pillar mapping, automated findings, and AI-driven recommendations.

On-PremMonthly subscription deployed inside the customer's own AWS environment. Designed for security-constrained accounts and enterprise requirements.

Outcome

The outcome.

I measured success through outcomes, not usage. A successful product outcome was a review that customers could trust and act on. I treated trust as a product feature, not an afterthought, addressing false positives, explainability of recommendations, and versioning of rules from the start.

Success metrics

Time saved per review. Replacing multi-week manual assessments with automated runs measured in hours.

Consistency of findings. Same environment, same output. Removed reviewer variability from the process.

Pillar coverage. Automated checks across all five Well-Architected pillars, not just the most common ones.

Actionability. Customers could act on findings directly, without needing a follow-up consulting engagement.

All Projects

GenAI Model Evaluation Tool