Agile in Data Driven (BI) Environment Rob Hoover

Agile in Data Driven (BI) Environment Rob Hoover

Who am I? Agile Coach 20 years Financial Services / recently moved over to Telecom. Agile since 2003 spanning Scrum, XP, Kanban, and SAFe. Experience in QA, Product Management, Data Operations, ScrumMaster and Agile Training & Coaching. Currently at TWC: RTE of the Residential Self Service BI Agile Release Train, Resi Agile Coach, assist all BI. Have some certs: CSM, CSPO, CSP, SPC4

Typical (non BI) Transformational Scenario Simplified Agile Coach walks into a shop. Says you have to deliver potentially shippable user value every two weeks. Team says “it’s different here.” Coach / team finds way to move forward, get to a good point. Typically, a successful transformation and a good team will figure this out, in many web based or OLTP system. However, for DW / BI / big data shops

Can you deliver (a full slice) of all this in two weeks?! User’s over here

Does that mean we can’t be Agile? No! So what do we do? Go back to the values and principles. Leverage some patterns used in the industry. Experiment / Innovate. Realize it’s ok not to deliver “an entire vertical slice” much of the time for sprint (start where you are!).

Avoid Ralph Hughes’ “231 Swamp” Refers to your typical bad / late waterfall data warehouse project. 2 years 3 million dollars 1st report For various reasons, BI / DW feels about ten years behind regarding Agile. So, if delivery takes longer than two weeks, maybe that’s not so bad?

Typical DW/BI User Story As a property policy rate analyst, I want to be able to compare total exposure to policy values for our top 10% of loss incidents, with drilldown by product category, customer demographics, and geography, so that we can identify clusters of customers for whom we should stop discounting property policies. - From Agile Data Warehousing for the Enterprise by Ralph Hughes

Resulting Changes ( 60 “stories”)

So, what are the goals? Take a bite of the elephant. A big enough bite to Provide value (to the customer, or worst case to the development process). Enable feedback. Small enough bite to reasonably finish in a given sprint (or acceptable / desired cycle time). Start where you are- the elephant metaphor applies to both the product and the process.

So, what are the goals? A large (if not the entire) vertical slice is still the goal, and sometimes attainable. For example, bring one attribute through the entire BI stack. Sometimes we will only be able to work at one layer (e.g. sourcing new data) in a given sprint. Often we will try to do more than one architectural layer in a sprint, especially if the business scope is small enough (e.g. operational system makes a view change for a consuming mart- code / test both the view and mart within a sprint).

How? First and Foremost - Be Agile

Techniques - Disclaimer I haven’t tried many of these- this is a collation of experiences and noted patterns in the industry.

Techniques (Traditional) Typical Agile / Scrum / XP Stuff. Top down Agile Culture / Support. Check in code regularly. Automated Testing / CI / TDD / Refactoring / Pair Programming. Embrace YAGNI (Ya Ain’t Gonna Need it). Automated Deployments. T-shaped People. High Value / High Risk First.

Techniques – Stories Fitting within Sprints Developer Story – This story will feed other stories before hitting the user. E.g. Sourcing Story, ETL story, Reporting Story. (Hopefully) Still be able to demo to a user for feedback. Design / Build / Test in Same Story/Sprint.

Techniques – Stories Fitting within Sprints Some options when your typical big stories can’t complete within the 34 day recommendation. Longer Sprints (Last Resort). Kanban “Look ahead” Modeling.

Innovative Techniques DevOps. Continuous Integration / Delivery / Deployment. A/B Testing. Build a sandbox for users. Experiment / Fail / Succeed / Repeat

BI Design Techniques Take one dimension and its attributes all the way through all layers (could be more than 1 sprint). Pull one field all the way from source to target.

Other Techniques As seen in Agile Data Warehousing for the Enterprise by Ralph Hughes: Hyper normalization data modeling- for designing a lightly integrated, persistent staging area for data from a company’s operational systems into a data warehouse. Hyper generalized data models- place all EDW dimensional data in an associative data store requiring only six logical tables, with measure data housed in lightly dimensionalized transaction tables.

Techniques – Follow the Innovators What are these guys doing?

Current Experience (Good) First end to end large enterprise DW/BI experience. SAFe can be helpful, with the right people in the right place. Several things lend themselves well to BI: Stories – Features. Shared Program Level People (Architects / Data Modelers). Product Owner / Product Manager structure and PI (Big Room) Planning lends itself well to plethora of stakeholders. Definition of Commitment.

Current Experience (Challenges) Too much stress on the team and chance for failed sprints: Most stories open within first few days of Sprint. Most close within last few days of Sprint. No Continuous Integration. Interruptions (lots of L3 support) / Merger. Shared Environments / Locked objects. Big Data environments (esp. prod like) are expensive. Floor the gas / Merger.

Helpful Resources Ralph Hughes’ books on Amazon (2/3 are on Safari online). Story Splitting: Bill Wake. Richard Lawrence. Lynn Winterboer. TDWI (People / Blogs / Conferences). http://agiledata.org/ (Scott Ambler). Martin Fowler’s Evolutionary Database Design. Petr Podrouzek’s LinkedIn Agile BI Series. Youtube has some videos (search for “Agile Data Warehouse” or some of the above entities / people).

What’s Next? Build a pattern repository (a la ScrumPLoP). Hear from the Group. Questions? What have you seen work? What suggestions do you have?