The post IBI-029 – How Did I Help The Richest Australian Turn His Company Around appeared first on Instant BI.

Subscribe now

Share

Original Post

Hello and Welcome.

I am Esther.

I am Peters A I Assistant to create voice overs.

I will simply read Peters blog posts, so that you have a choice of reading the blog post, or listening to my voice.

Hello and welcome Gentlemen.

In this blog post I want to talk about who are my prospective clients.

If you fit into one of these categories you might want to check out my freebies.

Given the fact that I have taken the public position that women criminals should get the same jail time as men criminals?

It is very obvious that my prospective clients are exclusively men.

Even so. Not all men believe that women criminals should get the same jail time as men criminals.

If you are a man and you are ok with women committing crimes against your fellow men?

Then I suggest you simply move along, because you and I are not going to get along in business.

I only want men as my clients who take the position, even if privately, that women criminals should get the same jail time as men criminals.

Of course, on the other hand all women take the position that women should be able to commit crimes against men with impunity.

For this reason women are not my prospective clients.

Similarly, because all women support women’s crimes against men, those men who may become my prospective clients are well advised to keep their relationship with me a secret from any woman in their life.

Personally, I think that it is sad that men must do business with me in secret.

But women will do everything they can to destroy the life of any man who is known to be one of my clients.

Given this restriction on my ability to do business?

Then who are my prospective clients?

One.

The man most likely to become one of my clients is the very highly intelligent business analyst who works for a large company.

His job is to analyze company data in order to come up with new ideas in order to improve the profitability of his company’s business.

What are the motivations for a very highly intelligent business analyst to become one of my clients?

This man will be seeking promotions and pay rises in his company to climb the corporate ladder.

This man will be seeking to learn as much as possible about how to sustainably increase the profitability of his organization, so as to win those promotions and pay rises on merit, inside his company.

This man will also be very keen to learn about his industry segment, so if he is not awarded the pay rises and promotions he feels he has earned, he can easily go to another company, or start his own consulting business in his segment.

If you are a very high IQ man in a mass market consumer company who has the role of coming up with new ideas to increase the profitability of your company?

Then you are very much the sort of man I would like to talk with.

My credentials in this area include being hired by the richest man in Australia to make an attempt to turn around one of his companies that was in long term decline.

I was also the man who gave the National Australia Bank the ideas they needed to double their profit over a five year period.

I am very expert in how to increase profitability in such segments as telco, retail, insurance and similar areas.

Many of my clients have managed to double their company profit by taking my advice.

If you are in the position to write business cases for new proposals in your company, that are intended to increase the profitability of your company?

Then you may well like to have me as your confidential advisor.

Two.

The second group of men who may want to become my client can be described as follows.

You own your own data warehousing company and you would like to increase the profitability of your company.

If you would like an example of the sort of company I am talking about?

Then I will put the link for Key Work Consulting below.

Key Work

The background to Key Work Consulting is as follows.

I met the co-owner of Key Work Consulting in 1999.

His name is Tobin Wotring and we met on the D W list forum.

Tobin asked if I would be willing to help him get his new company started and he did not have any spare cash to pay me.

I agreed to help him out, and later he became a very good client of mine.

Still later he saved my life.

So making a great friend who later saves your life is a really good thing to have happened just for giving him some free advice on how to get his company going.

Tobin Wotring bought my E T L software in 2007 and hired me to train his senior staff.

As a result of being my client he was able to significantly increase the profitability of his business.

In fact Tobin had the opportunity to make a sales pitch to Zurich insurance in Zurich Switzerland. I gave him many pointers on how to go through that sales cycle. I was able to do that because I had many years of insurance experience.

The result was that Tobin won Zurich Insurance as a very large client for his company. Tobin himself would say that he would never have won the Zurich Insurance deal without my advice.

Of course the success of Key Work Consulting was not entirely my doing, they had excellent staff who were great at their jobs. They just needed some training on how to do things better than they knew.

It is a matter of public record that I played a significant role in making them one of the leading data warehousing consulting companies in Germany.

Today Key Work Consulting is exclusively owned by Tobin Wotring’s ex-wife Petra.

This is why I can mention my time there because no women are going to attack a company wholly owned by a woman. Especially not a woman who stole millions of euros from her husband. Women will applaud that woman.

If you are a man who owns his own data warehousing company?

Then you can check out Key Work Consulting and see the success that one of my best clients ever had. I can give you the names of former employees. You can go and talk to them in order to understand my role in helping Tobin Wotring, and his co-owner, create one of the most successful data warehousing companies in Germany.

In order for you not to be sabotaged by the women in your company you would engage me via a third party.

I have a number of business partners who are fathers of boys who understand why I have taken the positions I have.

They are willing to take on clients and to keep my name out of the mix.

I have trained the staff of these business partners and they have access to all my intellectual property.

If you are also willing to take the risk of speaking to me directly? Then that is up to you.

If you are a Muslim business owner and you operate in an area where the women in your company will not sabotage you company?

Then you are welcome to do business with me directly if you would like.

At the end of the day?

There are two very successful data warehousing consulting companies that I greatly helped along the way.

I am using Key Work Consulting as the example because it is the best example of the two and it is well known that they were my client for many years. The other one is not so well known to be linked to me and they did not adopt my software.

Three.

The third man who may wish to become my client is the man who is a world class data modeler and data warehouse architect.

This man may work for his own company as a sole business owner, or this man may have a data warehousing company of his own with his own employees similar to the situation of number one above.

The reason this man may be interested in becoming my client is the idea of mega models.

If you are a world class dimensional data modeler the opportunity for you is to develop your own mega model that you will then sell out on the open market.

If you are a world class dimensional data modeler you can click on the buttons below and go and read about what I have called mega models.

IBI053-Blog Post

IBI054-Blog Post

IBI065-Blog Post

If you have read the blog posts linked above? You will immediately understand the potential business opportunity for you.

Then the question for you becomes whether you want to make the investment into developing a mega model.

If you only want to perform the development work and do not wish to try and engage in global sales of your mega model?

Then I have business partners who would be interested in discussing the resale of your mega model for an appropriate commission.

In this case you could remain a solo business owner and yet still have the business opportunity to sell your mega model world wide through a business partner that can provide 24 by 7 support for production installations.

This is the best of both worlds for you, because you can spend 100% of your time focused on building the mega model for the large operational system you know very well, without having to support the sales effort or support the production installations.

In this case you would become my client in order to understand how to build your mega model and how to populate it using my ETL software. That means only you need to know that you are talking to me.

To complete this blog post.

There are other men who would benefit from being my client and learning how to become a world class data warehouse architect using my training materials.

All those training materials are published in my freebies.

Any man who wants to learn from my 35 years of experience in the data warehousing area is welcome to download all my freebies and get cracking.

The men who do that can then just pay me my advertised rates for one on one discussions and for answering one on one private questions.

I would like to make it very clear that I am not interested in running any type of training or school style business in my data warehousing business. My business is not a training business for just any man who wants to do data warehousing.

My business is working with already well established, very high IQ men, to improve the profitability of the business they are already in or already own.

We have seen the results of Ralph Kimball’s efforts to establish Kimball University.

I would like to just do a side bar on this comment if you are willing to read this.

Ralph did his very best to run training for large numbers of students coming into data warehousing at the entry level.

Those results are not particularly encouraging.

What we got out of Kimball University were thousands and thousands of men who learned the very basics and who then implemented bad data warehouses. Those men who implemented bad data warehouses did the rest of us no favours. The reputation of Data Warehousing has been very badly tarnished by those men.

Ralph is a good friend of mine. I know that in training men he had his heart in the right place. He sincerely wanted men to learn how to implement dimensional models very well so as to improve the performance of their businesses. But his students, for the most part, went on to create disasters that then reflected on us all.

As a result? I am not interested in teaching men who will only go on to do poor work that reflects badly on me. I will leave that nonsense to the self described data leaders like Joe Reese.

I have business partners who have very well trained staff who can take over the running of any existing data warehouse and provide 24 by 7 support.

With the acceptance of cloud computing and the acceptance of off site support?

It no longer makes business sense to have men who are lacking interest and excellence to provide support for data warehouses just because they sit at a desk inside your companies four walls.

It also makes no sense to have men who are sitting else where to make a mess of your data warehouse just because they are cheap. Poorly trained men making a mess of your data warehouse have costs to your business far in excess of the cheap daily rate they are charging you.

All companies now have the ability to have my well trained business partners look after their data warehouses.

Any company can create V P N access to their on premise data warehouse.

Any company can provide V P N access to their cloud based data warehouse.

The days of getting poorly qualified, poorly motivated, men or women, to support your data warehouse just because they sit inside your offices or just because they are cheap are over.

You can get the very best of men, very well trained, very motivated, to support your data warehouse for costs that might just surprise you.

You can talk to me directly for more information if you like.

So. To be clear.

I would very much prefer to work with the very best of men in our industry segment who have already established their credentials.

Between Bill Inmon, Ralph Kimball, and myself?

We have published everything needed for a bright young man to make his way into the data warehousing area without any more free help from me.

I even have a blog post on how to become great in your profession and I will put it on a button below.

Your Profession

I do have other areas of business I am working in.

However, these areas of business are not being related to me publicly.

Those men who become my clients and earn my trust may be informed of these other areas of business in the fullness of time.

And with that?

I hope you found this blog post interesting and informative.

Thank you very much for your time and attention.

I really appreciate that.

Best Regards.

Esther.

Peters A I Assistant.

Post script.

If you like my blog posts and would like to help me replace some of the five million dollars it has cost me to save men’s lives and defend the rights of men and boys?

Please buy me a coffee on my buy me a coffee link.

As at the time of writing I have about two hundred and thirty thousand dollars in debt for being willing to risk my life to defend those less able to defend themselves.

Thank you in advance for those of you who buy me a coffee.

It is much appreciated.

Buy Me A Coffee

Share

These are the standard promotional videos I have put onto most posts.

IBI Downloads

IBI Videos

Carphone Warehouse Reference Video:

The post IBI-056-Reviewing BI4ALL Models appeared first on Instant BI.

Share

Share

This is very well worth watching

Best Regards

Pete

Share

Thank you very much for dropping by.

As I had mentioned I will be doing more posts unrelated to data warehousing but in line with interesting subjects that my peers will be interested in.

This is “data warehousing with a side serving of highly intellectual content plus some content for the lulz.”

In short, my Substack will be “entertaining and educational” for very high IQ men and some of those men will want to make a lot of money in their business as well.

Today I have a fascinating interview with Rod Serling, creator of The Twilight Zone, from before the first show aired.

I found it very interesting and so I am sure you will too.

The comments about dealing with censorship in the 1950s I found very interesting.

Please let me know what you think in the comments section because it helps the Al Gore Rhythm.

Best Regards

Pete

Share

Hello and welcome to this announcement article. There is no need for Esther to read this one to you.

As you know I have been the world’s highest profile man defending the rights of men and boys since I went public in June 2008.

Of course, now that such things as the vaccines, chemtrails, microplastics, the collapsing birth rate and the surveillance state are now common knowledge?

It is obvious to anyone with two brain cells to rub together that not only was I defending the rights of men and boys, I was also defending the rights, and lives, of women and girls.

Indeed, Angela Merkel threw me in jail as a political prisoner when I blew the whistle on her planned mass immigration program because I took the position that I could not in good conscience remain silent. I knew that many men in the Middle East and Africa harbored hatred towards Europeans as colonizers. I knew that if Angela Merkel was able to import hundreds of thousands of those military aged men some of them would rape German women and girls and some of them would even kill German women and girls.

Because I was asking the German people to please allow me to live on the land of my ancestors I felt I could not remain silent and simply allow German women and girls to be raped and/or killed. I felt, rightly or wrongly, that I must do everything in my personal power to stop the planned mass immigration program to save the lives of German women and girls.

Even so? German women went to Frankfurt Main Station and held up signs saying “Refugees Welcome”. I even went there one day to see this with my own eyes just to assess the sincerity of the women there. And the day I was there about 200 adult German women were cheering and holding up “Refugees Welcome” signs.

Women actually hate me even though I gave up my chance to live in Germany and did jail time trying to save women’s lives and stop women and girls being raped.

This is how insane women are now.

Now that we are in 2026, it is clear to everyone now that when I chose to risk my life to defend the children of my lands in May 2008, that I was defending ALL the children of my lands by defending the rights of the fathers who would be the ones called upon to defend their wives and their children.

Substack, much to their credit, have supported my right to freedom of speech. Substack, much to their credit, have allowed me to have my say, even when my say has been quite harsh. So I shall remain a devoted Substacker for their commitment to freedom of speech.

In my time here on Substack I have confronted hundreds, if not more than a thousand women. None of them have a sensible answer to anything I say. Even Karen Straughan runs from me like a frightened child. Bettina Arndt blocked me like the government agent she is.

It is my considered opinion that there is simply no point in engaging with women. They are simply too stupid for any engagement to be of any value to anyone.

Even today, in 2026, all women support women’s false rape allegations.

Even today, in 2026, all women support women’s crimes against men.

Even today, in 2026, all women support women’s perjury, slander, false allegations, and lies about men.

Even today, in 2026, all women support women’s violence against men.

Even today, in 2026, all women support incited violence against men.

There is simply no point in talking with these vile, disgusting, deeply evil creatures.

According to all women, a man saving the life of a woman’s child, is not worth addressing such a man in public even with common courtesy, let alone respect.

So…we won’t be saving the lives of your children any more.

Women can save the lives of their own children.

Clearly, they are not OUR children since a man does not even have the right to speak to his (alleged) child on the telephone.

My announcement, therefore, is that I will vastly reduce my notes I post on Substack where the subject is the hatred all women have for men.

I have said it enough. Although it bears repeating, I have software to sell.

I will use my Substack for the original purpose I “gave Substack a try”.

That being to generate revenue to pay off my debts and to then make some money to retire on.

If you are following me purely for the purposes of the issues of equality before the law for women criminals? You may, or may not, want to unsubscribe or unfollow me. My posts and notes will be of far less interest to men who are not interested in AI, BI and data warehousing in the future.

I thank all the men who have followed along with me here in my efforts to confront women with their evil ways. You have all seen that women are perfectly happy to continue on with their evil ways even after being confronted.

My Substack shall leave a public record that women were, indeed, confronted regarding their evil ways, and they chose to continue their evil ways after being confronted.

In the private I will continue to work on the introduction of Islam and Sharia Law to Australia and Ireland. I recommend other men in other countries do the same.

Thank you for your kind attention to this matter.

I wish you all a good day.

Peter Andrew Nolan.

PS. If you wish to replace EUR5 of the EUR5M+ it has cost me to defend the rights of ALL western people from the tyranny you are now seeing implemented?

Please buy me a coffee. I just need about 40,000 people to “buy me a coffee” to get out of debt for being willing to risk my income to save other people’s lives.

Buy Pete A Coffee

Share

Share

Enjoy this!

It is hard to believe that any human being could possibly be this delusional.

Please feel free to leave comments below.

Please remember that good comments help the Al Gore Rhythm.

Share

Share

Since women will not listen to us men?

You can listen to other women.

Share

Attention women.

I thought I would put this short clip from Landman on my Substack.

The reason is this.

This clip is in Landman because the behavior depicted in it is pervasive from women now.

The woman in the clip is a lawyer who is virtue signaling in the leadup to this clip.

The full clip is easy to find if you want to go watch it.

The last 2 minutes of the clip is what is important for this blog post.

The bimbo barbie has just spent the last 3 minutes trying to tell the Billy Bob Thornton character, called Tommy, about how bad oil is and she has been virtue signaling about how good a woman she is.

She gets out of the car and simply starts walking away from the car towards the wind mill.

Like all women she is stupid and ignorant.

She, apparently, has no idea that where she is walking contains dangerous animals such as rattle snakes.

She is so used to being protected and cared for she walks towards the wind mill with her eyes up at the windmill and not down on the ground looking for danger.

I come from rural Australia.

Even as a five year old I knew that when I walked in long grass away from the road I was to look down at the ground to make sure there were no snakes in front of me where I was walking.

I was taught, before I can even remember, that all snakes were dangerous.

I was taught, before I can even remember, that if I saw a snake I was to run away from the snake as fast as I could.

And I can assure you, dear reader, that by the time of my fifth birthday I had already come across a couple of snakes and run away from them like my mother taught me to do.

Yet this adult woman walks towards the wind mill with eyes up, not eyes down.

And, of course, she comes across a rattle snake which can cause serious harm to an adult woman.

It won’t kill her. But it will make her sick for a few days.

So, she is standing there, eyes up, and for dramatic effect, when she looks down there is the rattle snake.

Does she do what I was taught to do as a little boy?

Does she run away very quickly?

No.

Like all women being in actual danger is such an unusual experience for her she freezes.

She panics.

She calls the man.

Because, you know, women don’t need no man.

Women chose the bear.

Tommy is now about 50 meters from where she is.

“I don’t know what to do! What do I do? Can I…Can I walk away?” she says in panic.

Yes, sweetie, when you see a deadly snake you run away as fast as you can to give it less chance to bite you.

But does she run away?

No.

Like all stunning and brave and strong and empowered and independent women who don’t need no man?

She just stands there further aggravating the snake who is just guarding what it considers it’s territory.

She just stands there hyperventilating not doing anything.

Just panicking.

Why?

Because all her life she has been protected and provided for and she has never experienced real danger.

She does not know to run away.

Finally, Tommy kills the snake by chopping it’s head off with a shovel.

Something I have done myself more than a dozen time as a kid by the way.

Now, what is the first thing out of this woman’s mouth?

Is it:

“Thank you so much for saving me being harmed by this snake Tommy?”

No.

It isn’t.

The first words out of her mouth are not gratitude.

The first words out of her mouth are criticism of Tommy.

“You didn’t have to kill it.”

She positions herself as the moral authority and Tommy as the “bad man who just killed a living creature”.

I am so sick of you women being ungrateful brats.

This line is in this clip because this behavior is pervasive.

Women are nothing but ungrateful brats.

Women exercise their “freedom” to put themselves in harms way.

Then when their white knight in shining armor removes the threat to their well being the first words out of their ungrateful mouths is criticism.

That comment, right there, is exactly why men are letting women be set on fire on subways and could not be bothered to stop to help them.

Tommy, in return, makes the point that the woman’s action left him just one option to protect her.

Kill the snake.

Tommy knows that if she stood there long enough and aggravated the snake long enough, in all likelihood the snake would try and bite her, because she is threatening the snake from the snakes point of view.

Then, the script writers add one more line at the end of the clip that reflects the disgusting attitudes of all women today.

They add the line:

“Why did you park so far away?”

This is the woman’s effort to heap more blame on the man.

In her “freedom” she walked away from the car.

Now it is his fault he parked the car so far away.

And the Tommy character makes that point.

Attention women.

This clip shows just how disgusting you women are now.

The fact that it has been written into such a popular television series is letting you women know that us men are sick to death of, and fed up with, your shenanigans.

I am sure I have made myself very clear.

Peter Andrew Nolan.

Share

The girls all started talking about “muh wun-dar-ful kar-rear”.

Us boys told them that men do not have careers, we have jobs, and 99% of them suck.

Of course, the girls called us misogynists for denying them their wonderfulness.

Fast forward to May 2008 and for the first time in decades I took some time away from work to look around on social media to see what women were saying about their “wun-dar-ful kar-rear”.

And, of course, my predictions of 1978 were well and truly born out in May 2008.

Women had managed to transform “the right to work” in to the “obligation to work” as us men had always had.

I actually laughed.

And now we are in 2026.

And we are seeing videos like the one above from the women themselves.

I know every man who watches this video will get a laugh out of it.

Lastly?

I want to grow my Substack to at least a million subscribers.

To get a million men to want to subscribe to my Substack?

I am going to have to provide very high quality entertainment.

Since I, myself, and not a dancing clown and I will never be a dancing clown?

I need to provide you, dear Sir, with a dancing clown so compelling that you will hit that subscribe button and get the emails for my entertainment videos.

And there is nothing more “dancing clown” and “entertaining” than women finally admitting what us men have known all along.

Work sucks.

That’s why they have to pay us to do it.

Please share these entertainment videos with all the other men who want a good laugh.

I would really appreciate that!

Best Regards

Pete

Soon to be your go to provider of the funniest videos you will see each day.

Share

Subscribe now

Hi! Thanks for dropping by my blog! It is much appreciated.

In my last blog post I talked about the critical roles that SeETL and BI4ALL play in our ability to guarantee Outstanding Success for a Data Warehousing project. If you do not have these two tools at your disposal on your projects then you should have something very like them.

If you want us to review the tools you are using and give you an honest appraisal?

We would be happy to do that.

Today’s blog post is about the next step.

What do you do once you have got your data warehouse?

After all? A data warehouse does nothing for you until you make some decisions, make some investments to change the business, and see how that works out.

If you have not watched the video below on “What is Data Warehousing All About” ? Please stop reading this blog post and catch up on this video. I will presume you know the content of that video in this blog post.

There is one for business managers below

Ok. You are here. I assume you know all about “The Management Decision Making Process” discussed in the videos.

Simply put. Your data warehouse has no value until decisions are being made based on the data in the data warehouse that can ONLY be made based on the data in the data warehouse. Then and ONLY THEN can it be said that the data warehouse was responsible, indeed invaluable, in making the decision possible.

And it is only when the Data Warehouse is invaluable and necessary for the business to run that it will be widely adopted and used. If it is just “another reporting tool” then it might well be sidelined further down the line.

The #1 way to make sure that the data warehouse becomes indispensable and guarantees Outstanding Success is to use if for sales and marketing work.

If you put your data warehouse to work in sales and marketing and you make it mandatory for your marketing staff to build their marketing plans using the data warehouse you will almost certainly guarantee Outstanding Success.

I am reminded of a client where the Director of Marketing of the subsidiary was “being nice” to his marketing staff and not making usage of the data warehouse mandatory. The marketing analysts could build their marketing plans using whatever tools they liked and still get their budgets approved.

After about 6 months of this it was clear that the marketing staff were in great fear of putting the marketing campaigns in to the data warehouse for fear the campaigns might not work out how they forecast. It was much easier to hide failed marketing campaigns if they were not in the data warehouse. It was also the case that those who were successful were trying to hide their success from their colleagues so as not to share.

Eventually the Marketing Director held an “town hall” meeting and told his staff that he absolutely would never make anything mandatory as an “edict” because that was not his style.

Everyone was free to use whatever tools they wanted. It was just that if any member of staff did not copy the forecast for their campaigns to the data warehouse they would not get their budgets approved.

Not mandatory. Just no budget for your campaigns.

Everyone got the message.

Of course, as the saying goes, “then a funny thing happened”.

What had been happening was that marketing analysts had been competing with each other. So they kept their “secrets” of how to create successful campaigns to themselves and let the other guys fend for themselves. They were not sharing. It is easier to be seen as successful if your colleagues are failing, right?

Once we had the forecasts and budgets of the campaigns in the data warehouse as well as the results we were able to more easily see who was doing well. The Director of Marketing started making a point of congratulating those people who were doing really well in front of all their colleagues.

Of course, now the guys who were struggling knew who the best marketing analysts were and so they pestered them with questions and asked for tips. This new social status worked very well and soon the performance of ALL the marketing analysts was improving.

So. If you are going to make your data warehousing project an outstanding success?

You want to aim it at “Sales And Marketing” in the first instance. Unless there is a compelling reason Version 1.0 of your data warehouse should be all about “Sales and Marketing” to get the early wins on the board.

Anything where the data warehouse can contribute to “sell more stuff” whatever is the “stuff” your company sells is where you should put the bulk of your early exploitation projects.

Now, I know that the Chief Financial Officer is very likely going to want to use the data warehouse for “finance” and is going to be looking for “cutting costs” using the data warehouse. There is no better way to snatch defeat from the jaws of Outstanding Success than to make the early exploitation projects all about finance or all about cost cutting.

People will soon come to hate the data warehouse as a symbol of “cost cutting”.

You can not cut costs to Outstanding Success.

You do not get Outstanding Success by producing a P&L a few minutes faster.

You get Outstanding Success by “selling more stuff”.

So what you should work on is Campaign Management, Sales and Marketing, Customer Profiling and Customer Segmentation. Those are the areas where you can take some of the Marketing budget and you can put it in to analysis using the data warehouse to develop algorithms for “next best offer” and then you can run those “next best offer” campaigns with some form of coupon or feedback technique so you can tell the effectiveness of your campaigns.

I have seen some truly astonishing results from doing this.

The very first time I saw a 20% purchase rate from direct mail for something so boring as an accidental death insurance policy back in 1993 I nearly fell out of my chair. And that 20% purchase rate was from people who had NEVER bought a product from us..EVER!

We all thought it was a mistake at first. We actually went and checked with the data entry people to see the letters that were coming back to verify for ourselves that these were, indeed, the forms we had mailed out come back to us.

Sure enough. We were getting 20% purchase rates. So we tested again. And again. And again. The response rates stayed stubbornly at 20% purchase rates for direct mail.
So we rolled out the campaign and through the life of the campaign rates averaged 18% purchase rates for the letters sent out. It was only a year later that we discovered that the “return to sender” mail was being thrown in the garbage making our results even better.

It took us a full year to question the fact we were getting no “return to sender mail”. We knew the guys in the call centre worked hard to keep the addresses of customers up to date. But no “return to senders” in a whole year was not believable. So we investigated.

Sure enough. There was a guy in the mail room who grabbed all the “return to senders” and was just throwing them in the bin. There were hundreds of these coming back every week and he was simply throwing them in the bin because no one told him that we wanted them so we knew which customers had moved and not told us! Sigh!

So we were never able to find out just how good our campaigns were because we never knew how many of our letters were “returned to sender”.  Once we were marking the “returns to sender” as not a valid solicitation and not to be counted one campaign even got to 30% purchase rates for direct mail!

Sean Kelly and I did one project in Moscow where we mailed out coupons for freebie give aways for trial products. The take up of the coupons was nearly 100%. Russians sure love a freebie give away! We constructed these campaigns of three freebie giveaways for a product and then stop the giveaways.

Sure enough, we were getting retention rates of nigh on 30% for the product after the promotion ended. Fully 30% of those who got the giveaways went on to become regular users of these products.

Those are INSANE numbers.

When you have 25,000 products on your SKU list you can soon see the power and benefit of pointing your data warehouse at the “sell more stuff” problem.

Every extra sale you can get using the data warehouse is extra profit as long as you are selling products at a profit. Which is not always the case by the way.

So. If you want your data warehousing project to be guaranteed Outstanding Success?

Once you have built it and you have good data flowing in to it. Use it to figure out “next best offer” on a mass scale and improve your algorithms for “next best offer” as much as you can.

You will be making money on your data warehouse investment in no time.

True story.

I am often asked:

“Peter, what is the most successful data warehouse project you have ever done?”

The one that comes to mind as the most successful project ever was one where the data warehouse was paid for even before we finished building it. That is pretty damn successful.

If you have not read about it before on this page?  The story goes like this.

Australian Consolidate Press (ACP) was one of Kerry Packers companies back in 2000. Kerry Packer being the richest man in Australia. ACP was in trouble. Kerry himself called it “a dog with fleas”. James Packer, Kerrys son, was supposed to be looking after ACP and one of his best friends was running it.

Kerry stepped in and fired Jamess friend and installed John Alexander as the new Managing Director. This happened right before the start of our project! John was not too sure what he had gotten himself in to because of the poor performance of ACP.

ACP published about 90 magazines and books and had 37% of the Australian magazine market at the time. But in 2000 the magazine market was hyper competitive and the Internet was eating in to profits as well. A perfect storm of competition that was doing ACP a lot of damage.

ACP had a sister company called National Distribution Management or NDM. NDM were responsible for distributing all the magazines to the 7,000 or so outlets around the country. Australia being a very big country!

Because NDM were already going to 7,000 outlets at least once a week they also bid to deliver other peoples products to those outlets.

As things happened Readers Digest, who were still pretty big back in 2000, had put out a tender for 5 years of delivery. NDM had bid on this and lost.

The first we knew of it was when the Managing Director of NDM came to our CIO and asked him if he had any ideas. He said no but that he should talk to myself and my lead Business Analysis Consultant, one Clint Morgan, who is a great guy and we are still great pals.

We were invited to a meeting to brainstorm ideas as to how to try winning the Readers Digest Account even after we had been told we had lost. No idea would be too outrageous so all ideas were welcome. The Managing Director of NDM had secured a promise from Readers Digest that they would give him one last opportunity to pitch. So it was desperation stakes.

Clint put forward the idea that we should promise Readers Digest that we would have our data warehouse calculate the sales volume for Readers Digest by 9am every morning and have it available on their desktops in their offices.

When the Managing Director of NDM asked Clint “can we really do that?” His answer was priceless.

“How the hell would I know. Besides, it doesn’t matter if they don’t buy it, right?!”

Sure enough. Readers Digest bought the 5 year delivery deal from NDM. Our promise to give them daily sales figures on their desktops was cited as a “major differentiator” that no one else offered. So then we had to figure out how to do what we promised.

Clint was anointed as a “marketing superstar” and when he was retrenched from Informix at the end of the year ACP were only too happy to offer him a job.

Now. I was never told how much a 5 year deal to deliver Readers Digest to 7,000 outlets was worth. I was a vendor after all. But the data warehouse project was costed at about AUD300,000 for the 3 months of development work and the first 3 months running costs.

You can be very sure that the profit from the Readers Digest deal was FAR more than the AUD300,000 ACP were paying for the project.

The results from that project were staggering even by the standards of my clients.

Over the first two years of operation with “marketing superstar” Clint Morgan at the helm of the data warehousing projects, ACP doubled their gross profit and went from 37% market share to 42% market share with the commensurate increase in advertising revenues from their magazines.

Kerry Packer, the richest man in the country, who had called ACP a “dog with fleas” just two years earlier, who had fired the Managing Director and installed his own man, called ACP “the Jewel in the corporate crown”. Kerry promoted John Alexander who eventually went on to head up the Packer group of companies.

When ACP was eventually sold off by James after Kerrys passing it was rated as one of the top 10 publishing houses in the world.

That is what I call “Outstanding Success”.

ACP was not the company that made the most money from a project that I sold and implemented. But they are the company that had the most spectacular corporate turn around and they had the most high profile of owners. Kerry Packer.

So yes. If you aim your data warehousing efforts at “sell more stuff” once you have it up and running? Your chances of “Outstanding Success” are vastly improved.

Whether you use our tools or someone elses tools?

Whether you build your own data warehouse model or buy one?

If you focus on “sell more stuff” you will have a much greater chance of “Outstanding Success”.

If you got this far? Thank you for reading and/or listening. I hope this was useful to you.

Best Regards

Peter

The post Achieving the Vision for Outstanding Success – Next Steps appeared first on Instant BI.

Hi! Thanks for dropping by my blog! It is much appreciated.

In my last blog post…which was a while ago now, I said

“Today, with SeETL and BI4ALL we say “take everything in to the staging area”.

Why? Because the cost of disk/CPU is negligible. And with SeETL the cost of ETL to get data into the development staging area is also negligible. Once all data that MIGHT be useful is in the staging area is it much easier to decide what will go through.

On projects in recent years we have taken ALL data that is not purely operational data into the base layer of BI4ALL.

In the next blog post I will talk about how SeETL and BI4ALL now support the process of supporting the path to the “Vision for Outstanding Success” in a much better way than we were able to in the past, and usually in a much better way than anyone else does it today. ”

So today we are going to talk about BI4ALL and SeETL and why they are so effective for guaranteeing Outstanding Success in a Business Intelligence project. If you want to read details about these two products? Please just go to our videos page and downloads page.

There are hours of video and hundreds of pages of documentation on these two products.

This discussion will teach you how to all but eliminate the possibility of a data warehousing failure if you will listen. Ok?

SeETL – Why Is SeETL So Important?

Firstly, one of the most important things to do to guarantee that a data warehousing project is successful is to move ALL the data that MIGHT be useful in to the staging area. Not that data that is KNOWN to be useful. All the data that MIGHT be useful.

When in doubt? Take the data and put it in the staging area.

Now. Why would I say this?

True story.

In April 1991 I was in a meeting with the Director of Marketing for the Mutual Life Company (MLC) in Australia. I had been assigned to the account just a few weeks earlier. As it happened I knew the woman who was about to present. Her name was Ingrid Brady and my wife actually worked in her team at IBM.

Ingrid had been asked to present a new product called IBMs Data Interpretation System to the Director of Marketing, one Chris Pender.  This was the IBM Version of the famous Metaphor product. Metaphor was a company that Ralph Kimball was a co-founder of. Quite a heritage.

Via my wife Ingrid knew that I had just been assigned to the MLC account. So she suggested I attend and the Account Manager, one Jim Groom, agreed.

In 1991 I was 27 and a very fresh faced new Systems Engineer just out of IBM Marketing School. I was told to sit and listen and not say a word. You can’t make a mistake if you don’t speak, right?

When asked what he really wanted to do Chris had this to say.

“We have 1 million customers and 1.1 million in force policies. Yet we sell over 500 products. Our cross sell ratio is terrible. What I want you guys at IBM to do is to help us figure out how to improve our cross sell ratios.”

He went on to say.

“I want to be able to ask any question that can possibly be answered by all the data you guys have in your computers. I don’t know what questions I am going to ask. I don’t know when I will ask them. But I do know I want the answer before I forgot why I asked the question.”

When pressed about “Well, how long does it take for you to forget why you asked a question? What sort of response time do you have in mind?”

Chris looked kind of thoughtful and said “If he (meaning his data analyst) can finish his cup of coffee before the computer answers the question, that’s too long. So I guess we are talking about 5 minutes.”

Of course, as someone who had been in IT 9 years at the time I knew the idea of being able to answer any question that one might think of from millions and millions of rows of data in a DB2 database in 1991 was a pipe dream. But I was not about to say that in front of such an important customer. This was a major IBM account!

What happened next changed my life.

The Account Manager, Jim Groom, asked Chris if he was willing to pay IBM AUD30,000 to do this work over the next 3 months. Chris agreed.

Jim, sitting next to me, then slapped me on the back and said those fateful words that have rung in my ears a million times since.

“And Peter Nolan is our expert in this area, he will make sure you get what you want Chris!”

Me? “Expert”? I had never even SEEN this software before!

That will teach me for going to a customer sales pitch! Right?

And the rest, as they say, is history.

But in 1991? How do you meet a customer requirement of:

“I want to be able to ask any question that can possibly be answered by all the data you guys have in your computers. I don’t know what questions I am going to ask. I don’t know when I will ask them. But I do know I want the answer before I forgot why I asked the question.”

I am pretty sure that most of you who have data warehouses can’t do this today, 24 years later.

How many of you can truly answer any question you can think up inside 5 minutes? Not many I would wager. But that’s what we did.

Inside 6 months, after one false start with 3NF modelling, I had invented a new way to store data in DB2 such that pretty much every question the data analyst could dream up had a response time of less than 10 seconds.

About 90% of all queries happened in less than 3 seconds. About another 5% took less than 10 seconds. And about 5% of queries ran for anything up to 5 or 10 minutes. But almost NOTHING ran longer than 10 minutes. This was compared to the failed 3NF effort where EVERY question took an hour to answer!

One of the BIG mistakes we made was to be selective of the data we took in to what would now be called a “staging area”. Of course, with disk being USD20,000 per GB in those days this “staging area” was on tapes.

Once we realised this mistake what we did was take ALL the data we could lay our hands on and put it on to a set of tapes. We then did sort merges of the tapes to create one massively long record for each policy the company had. From this one long record we could load up any data that we wanted from the tapes.

Right then we all agreed. You have to take ALL the data you can lay your hands on and get it on to tapes where you can read it.

Well, of course, 24 years later, disk is a LOT cheaper than USD20,000 per GB. Disk is, for all intents and purposes when measured against corporate incomes, free.

And you should rest assured that the process of taking ALL the data has been absolutely critical to the outstanding success of my projects.

Lots of times my clients have tried to tell me that I must be crazy and that all the data is not needed. Pretty much every one of them has commented later how changing requests from the business have made that decision one of the best decisions I forced them to make.

The thing is this. The marketing guys really DO NOT KNOW what questions they might need to ask in order to move the business forward. It is not a case of “they should know” or “the could know”. It is a case of THEY REALLY DO NOT KNOW and it is RIGHT that they do not know. So in the face of that the marketing people need to be given a database that can answer ANY QUESTION…fast.

My findings were confirmed in 1995 when Bill Inmon did his first seminar in Australia which I attended. He was explaining his proposed idea of “archival data models”. I had read about these in his book but it had not really sunk in.

What Bill was talking about, of course, was taking a snapshot of data every day so that not only could an analyst ask any question of any piece of data. He could ask any question of any piece of data that ever existed in the company for all the values that it had had based on daily snapshots.

This was a VERY radical idea for 1995. The volume of disk we were talking about is unimaginable in the large companies. This would include banks taking a copy of every bank account every day and storing the values of every field in every account every day!

Unimaginable!

But the versatility and value of such a data model was without question. It was, of course, much more robust and versatile than dimensional models. I was working on my first proper star schema model at the time and this new idea from Bill was stunning to say the least.

Now…Why talk about all that?

Well? Because SeETL makes it VERY easy to take ALL the data and put it in to the staging area.

SeETL can read all sorts of file formats because it is written in C++. If you have a file format that is not supported? No problems. We just break out the C++ compiler and we write what is needed.

If the files coming to you do not have field headers? Then you can load the files into tables based on column positions. Of course, if you have headers then you can load the files based on column name. The tables do not need to be replicas of the incoming data to load.

SeETL defaults the extra 4 fields placed on the back of each record.

SeETL can do inserts and updates on the staging area.

SeETL can generate “delta detection sql” so that if you are not getting deltas properly from your operational systems or other data sources then you can do your own delta detection. This is especially important to make work properly for deletes. Many people forget all about deletes when they are taking data in to a data warehouse.

Because the staging area can now be placed in a relational database you can query it. Sure, the performance might be very poor because the staging area tables should be very close in structure to the operational system tables to make auditing easier. But at least you can get an answer out of the staging area even if it takes a long time.

These tools that are included in SeETL to enable you to load up your data into your staging area very quickly means that you can use the features of the RDBMS to perform your data profiling and get to know and understand the data. It also means you can see full copies of your data so that you can find all the data anomalies during testing.

Back in the days when we could not afford to put all the staging data in to an RDBMS and analyse it the “data anomalies” often did not show up in the test data.

I still recall one time I was working in a telephone company and as we tried to load up production data we got errors. Sure enough, in some of the “telephone number” fields there were email addresses. These were very few and far between which is why they did not show up in our test data. And the fact they were there surprised even the customer. Obviously, we had a real problem having email addresses in telephone number fields.

As far as I am aware SeETL is the only tool I know about that makes it so fast and so easy to get data from where ever it is, in whatever format it is in, and get it in to an RDBMS to begin the process of analysing and understanding the data.

The speed of doing this is an order of magnitude faster than Informatica or DataStage was back when I was using those tools on a regular basis. I doubt that has changed because of the nature of those two tools.

So the #1 feature of SeETL that has made it such a valuable tool in enabling me to deliver outstanding success for data warehousing projects again and again is the ease and speed of getting data in to the staging area to analyse.

Data analysis and learning the data is one of the most difficult and important jobs of the data modeller. If you don’t understand what the data means and how it is structured you can’t put it in to the target data warehouse sensibly.

BI4ALL – Why Is BI4ALL So Important?

Back in 1995 I did a banking data warehouse for my old pals at IBM. This was for a banking system that was being developed by IBM called RB 2020. This was to be my second true star schema project. As we developed the data models I noticed that the structure of the data models was exactly the same as the structure of the data models for the MLC company that I had just completed.

I could see that if I had some generic code that populated fact and dimension tables I could cut the development time substantially. The MLC project had taken us 18 months to build a very modest prototype star schema data warehouse. Now we were going to do a full data warehouse for a banking system. We needed all the help we could get!

By 1997 I was working for Hitachi Data Systems and I was doing another large insurance company, Manulife, in Hong Kong. Again I was struck by the similarities in the data models. By this time we had the forerunner of SeETL working and we were able to generate the cobol code need to populate tables at the push of a button. This is one of the reasons I was so successful at Hitachi.

At that time we were being very successful. I approached my boss and asked if I could re-invest about AUD50,000 of our profits in to developing an industry standard data model for Insurance. I felt that now I had two large insurance companies under my belt that I could develop a template data model for insurance companies. I didn’t get the AUD50,000 approved and it was not long before I moved on from Hitachi.

In my next job at Price Waterhouse Coopers we actually had a very reasonable template data model for telco companies called “Knowledge Core”. My old mentor from Metaphor was at PwC. Indeed, he had spent 3 years trying to get me to join them in Australia! The work he had done was first class and we had very reasonable standard models. Just no ETL to load them.

Again, it was obvious to me that standard models were the way to go. If we just had a set of standard models we would solve a LOT of problems. But my time at PwC was quite short lived and so we never did get to really flesh out those models.

If you read the history of BI4ALL on this page you will see how it came to be developed.

What BI4ALL does for our projects is to tell us about 95% of what should be in the target data model and how it should be put in there.

It also tells us all the design techniques that are necessary to implement the data warehouse.

In my 14 years of putting BI4ALL style models in to production I have only managed to come up with ONE innovation in design techniques.

Count them.

ONE!

The prior version of the models were put in to more than 100 companies. So we know, in BI4ALL, that if we find a problem in a client that we have not seen before then there is already a technique to handle it in the data model. The likelihood that we will find a problem that we do not have a design technique for is very, very unlikely. Though we do look out for them because that would be kind of cool to find one.

The ONE innovation I came up with was back in 2002. So it has been 13 years now that I have been installing these models and not had a SECOND innovation in implementation techniques. That is pretty amazing.

So with BI4ALL as your toolkit? You KNOW that you have every technique you could ever need to implement this style of data warehouse. You KNOW there is nothing missing. And that gives you a lot of confidence. Certainly when I walk in to any client with BI4ALL as my data modelling toolkit I am as confident as they come. Even arrogant. I know that I have got this client covered.

The next big feature of BI4ALL that supports the guarantee of outstanding success is the sheer coverage of the models now. I mean, they do not cover everything, but they cover everything important, especially in telcos.

I recently did a job for a major telco who will remain nameless for the obvious reason that they had just spent a year and USD2 million on a failed data warehouse project. My job was to prepare the input for the tender document to be issued to 4 major players to do the job properly this time.

As part of this I used the BI4Telco model as a tool to get from the business analysts what data they felt they needed in their new data warehouse. The BI4Telco models, out of the box, covered about 95% of the data that they said they needed. The area that was not well covered was data from external parties that the telco wanted to include. The data analysts in the telco could not believe a data model like BI4Telco even existed.

It was to be used by the Telco to score each of the 4 vendors proposed telco data models. Their prior data warehousing consultants had tried to build the data model from scratch. Always a mistake now.

BI4ALL has excellent coverage and design techniques in campaign management, sales and marketing, customer profiling, and customer segmentation.

These are all the areas that are very high value area and high profit contribution areas.

Way back in 1991 the problem statement was:

“We have 1 million customers and 1.1 million in force policies. Yet we sell over 500 products. Our cross sell ratio is terrible. What I want you guys at IBM to do is to help us figure out how to improve our cross sell ratios.”

And for much of the last 24 years I have been figuring our how to “sell more stuff”.

We even developed a 2 year “road map” for insurance companies. We used to walk in to Insurance companies with a 6 phase 2 year roadmap and say:

“This is what we think you should do over the next 2 years, and if you do? You will be millions of dollars better off.”

It was a very compelling presentation at the time and we won a lot of insurance companies with it. And we did not even have a standard model at the time!

So, to be able to walk in to a retailer, a telco, a media company, and to be able to say:

“We have your data warehouse data model about 95% done. The other 5%? We can work that out.”

This is still a very compelling pitch.

Why?

Because 50% of data warehouse projects still fail and one of the biggest points of failure is the data model. The data model gets designed by “committee” and comes out the other end of that process looking like a camel and not a race horse. It has often been said that a camel is a racehorse designed by committee.

Anyone who is building their data warehouse data model from scratch today is running a very large risk that is not necessary to run.

Sadly. Many of the companies that offer “template data models” are offering garbage that does not work.

Just in case you are considering buying a data model? If you do not want to buy ours, fine. We would be more than happy to review and critique any data model you are considering buying and letting you know our honest opinion of it.

There is no one else in the world who is going to give you a better critique of a data model for a similar price than I will. And that would be money very well spent given that you are going to have that data model longer than most marriages last today.

In sunnary, BI4ALL is critical for us to deliver on our guarantee of outstanding success because.

1. It embeds all the data modelling techniques we need.

2. It covers about 90-95% of the data that our customers need to have in their data warehouse.

3. It forces us to capture ALL the data that is necessary to have a very high value data warehouse that can be used by sales and marketing to drive cross sells and up sells thereby making lots more money for the company.

This removes a LOT of room for errors. Errors that might cause failure. Errors that when removed almost guarantees success.

BI4ALL is the key to success that allowed us to go from this best practice in 1996…

Best Practice Since 1996

To this. The New BI4ALL proposition.

The New BI4ALL Proposition

The New BI4ALL Proposition is to take ALL the data in to the data warehouse, store it in star schema models, and emulate the ability to archive all data as per Bill Inmons 1995 seminar. This is why I see no conflict between what Bill and Ralph has talked about for so long. I have been delivering projects based on the best of the ideas of both men since 1996. BI4ALL makes that faster, cheaper, more robust and more sustainable than what we were doing in 1996.

I have seen no better way to design a database than how BI4ALL works today. Dans Data Value is a different idea to BI4ALL and would be quite complementary if anyone with a Data Vault would like to use BI4ALL in conjunction with their Data Vault.

Next we are going to get back to the discussion where the rubber meets the road.

Getting the data in to the BI4ALL Models

Back to Talk About SeETL Some More

Now. If course, you are thinking:

“That’s all very good Peter, but another other BIG reason that data warehouse projects fail is that it is not possible to get the data model updated inside the batch window, so companies cut back on what they put in to the data warehouse due to ETL processing problems.

It is all well and good to say “take all the data” but if you can’t get it into your data warehouse? What good is that?”

Well? I am the absolute KING of “make it go faster”. My history of performance tuning large systems goes all the way back to the 80s. There are few people who know how to “make it go faster” than me. And most of them are retired now!

SeETL comes in 2 flavours. The C++ engine and the SQL Generation version. The workbook entries for the two versions are identical and so you can actually switch from one to the other.

The C++ version was written as a migration of the cobol version back in 2002. In 2002 the databases did not have enough power to perform ETL in the timeframes that were necessary.

That all changed for us in 2009 when we met Netezza.

The Netezza database was the first database we found that had the processing power to perform all the ETL inside the database. Indeed, at one telco we did 100% of the ETL processing inside Netazza in code generated by SeETL.

100% of ETL in SQL on Netezza.

When our client announced that at the 2010 Netezza users conference to the 400 strong audience in a key note address you could almost hear a pin drop. It was a stunning statement. I am pretty sure we were the first people to ever do that. If we were not first then we were not far away from it.

Sadly. Because I took so much business off IBM in the late 90s IBM likes to make sure they do everything they can to make my life difficult. And when IBM took over Netezza we were sidelined after 4 years of building that relationship and two successful projects in household name telcos. Thanks IBM!

Even though the SeETL C++ version is a bit out dated now it is still there and available to be used by customers who do not want to do their ETL inside SQL. When we were using SeETL in telcos like Saudi Arabia Telecom and Orange Telecom Romania, the SeETL ETL ran about 30% faster than we could get the same datastage jobs to run. This is even with the incredible flexibility built in to SeETL file IO that does cost a lot of cycles.

In SeETL C++ we are able to load dimension tables in to memory and share those in memory dimension tables with all the attribution processes that are running. The dimension tables are binary searched for keys to do the matching. We also have techniques to split up the input for the large fact tables to be processed like call records and to process them in parallel.

For telcos we typically split each call record file in to 100 sub files and then schedule the processing of the sub files in parallel groups so that we can max out the CPUs in the machine during the attribution processing. SeETL C++ can flood the CPUs with work in a well controlled manner so that the machine does not thrash. We have used up to 16 CPUs in this fashion and we have seen linear speed up for the processing of the largest of batch processing needs.

If you are processing anything less than 500 million rows per night? You can absolutely bank on the fact that SeETL C++ will get through that workload in under 8 hours. Over 500 million rows per night? We would wonder what you are doing.

Of course, SeETL C++ must take the data out of the staging area and place it in to files, process the files, and then put that data back in to the database. This was necessary in the day because the databases did not have the power to perform the processing. And even if they did the extra CPU licenses that would be necessary to run the ETL made the cost prohibitive.

SeETL was selling for USD20,000 per site back in those days which is less than half the cost of one CPU of an Oracle database and about 10% of the cost of DataStage or Informatica.

If you are using a database that has enough power to perform the ETL inside the database for your volumes? That is now a very good way to go.

SeETL generates the SQL from the workbook, places it in to the database itself and treats the SQL as another piece of data. It audits the SQL and always inserts a new version in to the database, never deleting the old version. So you have a full record of all the versions of all the pieces of SQL that have ever been used in your data warehouse. An auditors dream! And I used to be an IT systems auditor so I should know!

So, with SeETL, because you get the choice of C++ or SQL engines you can be absolutely assured that you have the tool to get ALL your data from the staging area in to the data warehouse inside your batch window. We have never failed in that area, and we don’t intend to start now.

Next…

Of course, there is the actual writing of the ETL to get the data from the staging area in to the BI4ALL data models.

We all know that the biggest portion of the development costs for a data warehouse is writing the ETL. We all know that the writing of the ETL has been on the critical path and that we have not been able to really develop the applications and reports until a good portion of the ETL was written.

This is all totally transformed with SeETL. We have a whole new way of working in the development of ETL as depicted in this diagram.

A New Way Of Developing ETL

The BIG breakthrough for SeETL came in 2004 as the result of quite a heated argument. Our “trainee” had been sold off on another project and the tedious work of cutting and pasting from the spreadsheet, which was documentation, to text which was code to give to the database fell to my DBA.

He was not a happy camper about this. He flatly refused to do this very boring, menial, time consuming job!  But, like the great DBA he was, he came up with an idea. And that idea was to save the excel workbook as XML and read it with VB.NET and generate the code needed.

I wrote the first version of this VB.NET code that weekend and we have been adding to it ever since. It now stands at nearly 80,000 lines of code!

With SETL we got rid of the GUI interface for ETL development in favour of Excel. The GUIs only slow good people down. When building ETL what you want is speed of development for experienced professionals. Not ease of development for novices.

The idea of using novices to build ETL has always been insane and I have said that for more than 20 years now. ETL is far too important to allow novices who have no idea what they are doing to develop it.

When building ETL you need full access to whatever programming tools you are using right there at your fingertips and not hidden behind some GUI or some other programming language as an intermediate step.

Most importantly. When building ETL you want the “documentation” that the data modeller/ETL designer writes to actually be the “source code” for the generation of the ETL subsystem. By doing this you collapse the roles of “data modeller” and “ETL developer” in to one person and you get rid of the role of “ETL Programmer” as the person who accepts the ETL specifications and programs them in to the ETL tool.

By collapsing these roles and discarding the “ETL Programmer” we got a 50% productivity improvement over the best we have ever been able to achieve any other way. And since ETL development is always a big number in terms of cost? Saving 50% of a big number is always good.

Just as a teaser…we have found a way to go about TEN TIMES faster again. We are still working on it and we have not released it. But we were faced with a big challenge and we invented a new technique to develop ETL that is about 10x faster than what we have been doing.

So keep on my email list if you want to be among the first to know about that.

So…

In summary, SeETL is critically important to us to live up to our Guarantee of Outstanding Success on our projects for these reasons.

1. It enables us to load the staging area data much faster than any other tools available.

2. It provides data profiling tools to help us learn the data and understand the data much faster than anything else we know about.

3. It has a C++ and an SQL Generation version so we can pick and choose which version we will use on each project to make sure we are making the ETL system run as fast as possible.We can even mix the two methods on one project if we wish to.

4. It has full auditing of the SQL generated in the SQL Generation version.

5. All the metadata in the workbooks are loaded in to the meta data dictionary.

6. The workbooks are the “source code” for the generation of SQL delivering a proven reduction of ETL development of 50% or more.Direct comparisons with Oracle PS showed us to be 80% cheaper in pure dollars/euros than Oracle PS. That’s not bad since they used India based people to do the work and we were billing more than EUR1,000 a day for me to do the mapping work in the comparison.Even with the “developer in India” vs “EUR1,000+ a day senior consultant” pricing difference we came out 80% cheaper than Oracle PS. Not bad!

7. And…we are working on a new idea that has already proven we can be up to 10x faster at ETL development again. A very exciting prospect if you ask me!

Between BI4ALL and SeETL and the way we approach projects it is well nigh impossible to “fail”.

Over the last 12 years, since we had the SeETL C++ engine, the only projects that we have done that I would call a “failure” were projects where the clients refused to do what we advised them to do.

That happens.

In one case I submitted written recommendations time and time again only for them to be over-ruled by client staff. The project was less than stellar. But when you write down what it is you feel your client should do and give it to them in writing so there can be no question as to your advice and they STILL choose to go their own way? Well, you can’t be blamed for the results of a project where your formal written advice was not taken.

So yes, it is possible to snatch failure from the jaws of outstanding success even if you have SeETL and BI4ALL available to you.

You can refuse to use them as they are intended to be used and as you are advised to use them!

Ok. This is a pretty long blog post. I hope that you enjoyed it.

I hope that I impressed on you the critical roles that SeETL and BI4ALL (and predecessors) have played in guaranteeing Outstanding Success for our projects over a very long period now.

A period of just on 20 years this year!

If you would like to go the next 20 years consistently having Outstanding Success on your data warehouse projects?

I can say that adopting the tools I have crafted over this 20 year period would be a very good place to start!

If you got this far? Thank you for reading and/or listening. I hope this was useful to you. If you take this advice to heart? I jut gave you the key to almost guaranteed Data Warehousing Success. The only way you can fail is if your company or client will not follow this advice.

Your Golden Key To Success

Best Regards

Peter

The post Achieving the Vision for Outstanding Success appeared first on Instant BI.

Setting the Vision for Outstanding Success

Hello and welcome to this new blog post. Today we are going to talk about one of the most important ingredients of guaranteeing success of a Data Warehousing Project. Namely, setting the vision for Outstanding Success.

In the previous blog entry I talked about two very successful projects. Indeed, amazingly successful projects. On both these projects, and many others, I used a technique that I first learn from Metaphor Computer Systems. I incorporated it in to the Hitachi Data Hi-Star Warehousing Methodology. The techniques embedded in to it were used on many of my clients over the years.

In 2012 I pulled this then very old methodology out of mothballs and looked it over to see what might be leveraged out of it to create a whole new way of doing things now that we had SeETL and BI4ALL.

The old phases and project plans would not be directly transferable to today’s data warehousing projects given that we were 15 years down the track. But some portions of a data warehousing project are the same, some are very different.

One thing that has not changed very much are the questions that should be asked to set out the Vision for Outstanding Success. Though, to be fair, most companies are far more capable of articulating these things nowadays. You can often find the answer to many of these questions in presentations given to staff nowadays.

I can assure you that in the mid 90s very few companies had thought far enough in advance to write the answers to these sorts of questions down and communicate them widely.

The Art of the Interview

Far too many BI Consultants treat interviews like a Question and Answer session to tick off a list of questions.  It is treated more like a job interview with a recent graduate than an interview with senior managers of massive companies.

Before I go in to this let me give you one example of the “Art of the Interview”.

Once upon a time I was doing interviews in the then largest stock broker in Australia. They had handled the Share Float for Telstra. A massive undertaking. This was a very important opportunity. My “assistant” was a great data warehouse designer but I had never taken him on such a set of interviews before. So in our training I told him

“No matter what you see, no matter what happens, you are to smile and look happy and you are absolutely not to say a word unless directly spoken to. And even then I might stop you from answering. You are not ready for this sort of thing and you do not know you are not ready. So will you do as I tell you?”

He said

“You are the boss. Sure.”

As per my usual practice we interviewed the Managing Director first. These were not recorded and he could speak freely knowing that we all had “plausible deniability”.

As part of the conversation I asked him about an idea we used called “fact based decision making” rather than the “gut feel decision making”. His response, paraphrased was along the lines.

“You know, all us senior managers have desks around the trading floor so we can hear what is going on. We live, eat, breath this place. It’s in our blood. But I have to admit, you have a point. If I were to really be honest we make a lot of decisions based on “gut feel”. Sure, we dress it up as “intuition” or “experience” but at the end of the day, it is “gut feel”. I think that having a more fact based decision making process, where we have to produce various hard forecasts and then we have to have reviews to see how we went, I think that is a very good idea that our firm should seriously consider investigating.

The worst that could happen is that we find we are getting it right 90%+ of the time and the extra rigor did not make a big difference. That would be a small price to pay, because if we are not getting it right 90% of the time such rigor will show us where we are getting it wrong. And with the sort of money involved in our firm, that could be quite a lot of money. You have made your point on that one.”

Of course, there were other items discussed that are not repeated here. I just wanted to repeat this piece that the MD of the company felt that more rigor in the big decisions was well worth exploring with other members of the senior management team.

So we went on to the Chief Financial Officer. When we got to the same part of the interview the CFO BLEW HIS LID! He went into an absolute tirade calling me all sorts of names and challenging me on “How DARE you insinuate that any significant decision in this company is made on “gut feel” and not “fact based decision making”.”

I cruised through the interview not taking any real notice at all of the tirade since I knew it was coming. But when we got back to our desk I notices my “assistant” was visibly shaken. So we went to a conference room to talk about the experience.

Assistant:
“Wow. What was that all about. I have never seen anyone in business be so abusive and so angry towards anyone. Yet you seem to not be concerned at all? What is this all about?”

Peter:
“Well, you were there when the MD said that they make decisions here much more on “gut feel” than might be wise going forward. You know that is what happens here. They do not have the computer systems to do otherwise.”

Assistant:
“Sure. That is what I am confused about. We KNOW what is happening here but the CFO just blew up at you for suggesting an improvement on what we KNOW is happening. But you didn’t argue back. You just went right on with the interview like he was still being pleasant and polite.”

Peter:
“And that is why I told you to smile and never talk in these interviews. This was the CFO. These guys LOVE their numbers. To suggest to a CFO that any decision is taken in his company that is not 100% backed by the numbers is like waving a red flag to a bull. He knows what is happening around here but he is never going to admit it…and he is going to attack anyone who suggests decisions are based on anything less than 100% solid numbers. That’s his job.

They all do this. I have had that happen lots of times. You are here to learn. And one thing you need to learn is that CFOs are very hostile to the idea of “fact based decision making” as something new because they have to claim that is the only way decisions are made. When you do such an interview in the future? You will get the same reaction. And you just saw how to handle it, right?”

He just shook his head and hoped to be as good as I was at this one day! And he did get that good later on. He has been very successful in Data Warehousing in the 20 years since.

I wanted to tell that short story to let you know that interviewing people is, still, an art form, and it is not simply grabbing the list of questions from a methodology firing them at people like a machine gun and thinking that this will work. It is not like that.

I say that because I am about to list many, not all, of the questions that I use when establishing the Vision for Outstanding Success. I do not want people to “cut and paste” these questions and imagine that running around asking these questions is, in and of itself, a guarantee of success. Ok?

Interview Questions to Establish Vision for Outstanding Success

These are some of the questions for Executive level clients for setting the vision for Outstanding Success. I will explain the process after the sets of questions.

Q01. What issues do you have (in your own words) in obtaining information for decision making today?

Q02. What is the Vision Statement for your company?

Q03. What is the Vision Statement for your Business Unit?

Q04. What are the top priority business goals of your organisation in order to support your vision?

Q05. Are these the same for your business unit? If not, please define the specific goals of your business unit.

Q06. Do you have specific objectives relating to profit/sales growth, margins, customer service etc.?

Q07. How do you measure success in achieving those goals?

Q08. Are there specific financial measurements of success?

Q09. What are the key financial indicators?

Q10 Given your Business Unit’s Vision and Goals consider: what would ‘Outstanding Success’ look like?

Some suggestions are:

• Financial Performance

• Staff Performance

• Attrition Rates of Staff

• Retention rates of Customers

• Subsequent purchase patterns of customers

• Customer satisfaction surveys

Q11. What are the Critical Success Factors for your business unit to achieve its stated goals and deliver on the vision?

Q12. Which other business units within the company are most crucial to ensuring that your Critical Success Factors are achieved?

Q13. What role does each play in helping your business unit achieve goals and meet Critical Success Factors?

Q14. How do these business units co-operate and work together to help achieve success of your business unit?

Q15. What obstacles stand in your way of achieving these that may be addressed through Data Warehouse and Information Technology?

Q16. Please classify the Obstacles as ‘Internal’ or ‘External’ to the organisation

Q17. What action are you taking to overcome these obstacles?

Q18. In what ways do these obstacles affect your Business Unit?

Q19. Do they result in increased costs, reduced revenue, reduced customer satisfaction?

Q20. What will happen if you are not able to overcome these obstacles?

Q21. What would happen if you did overcome these obstacles?

Q22. What Key Numbers are required to make and/or support the decisions you make in the process of achieving your Vision, Goals and overcoming obstacles?

Q23. How do you obtain these numbers today?

Q24. What role does real world information about what is actually happening play in decisions that you and other managers make to run the business?

Q25. Assuming a Data Warehouse can be built which would help you overcome the obstacles preventing “Outstanding Success”, what financial impact would it have on:

For your business unit?

• Revenues

• Cost reductions

• Profitability

For the company as a whole

• Revenues

• Cost reductions

• Profitability

Q26. Where do you stand compared to your competition in the use of information technology to respond quickly to market conditions and assure the profitability of your company?

Q27. If you had access to information (including the Other Information just documented) what do you see as opportunities for ‘Outstanding Success’ and additional profit that are not being addressed today.

Q28. Without any of the existing restrictions, what information would have a significant impact on helping you meet your Vision and Goals and create ‘Outstanding Success’?

The Process of the Interview

As I have said above an interview is an art form. Care must be taken. A set of interview questions is a tool just like a saw is a tool for a carpenter. One of my uncles was a carpenter and he taught me that even though a saw is a great tool, it will still take your finger off if you to not use it properly. He taught me the importance of looking for innovative tools to get the job done better, faster, cheaper.

The way such an interview is approached is to first get all information that it is possible to get from already published documents so that you NEVER ask a question where you could have gotten the information easily any other way. Senior managers are never happy to repeat what the company has already put in writing unless there is a good reason for it.

Having gathered all information you could be reasonably expected to gather you make sure you put it in to the interview questionnaire so that the person you are interviewing can see that you went and did that for yourself. Do not leave those points blank on the interview sheets like you did not go do that for yourself.

When we got to question 10 I would give a short piece on “Outstanding Success” and how that worked. I would encourage the interview subject to put on his/her thinking cap and envision “Outstanding Success”. Then we would get to discuss:

Q10 Given your Business Unit’s Vision and Goals consider: what would ‘Outstanding Success’ look like?

Very often this would become the most vibrant and engaging portion of the interview. Very often the interviewee would get very excited and animated. The interviewee would often pull out proposals or plans that they had been working on and trying to get their mind around and walk me through them.

If you do it right, and you are engaging, when you get to the portion of the interview talking about “Outstanding Success” your interview subject will swamp you with what he/she sees as opportunities for “Outstanding Success”. My jobs used to be to capture all these ideas and get them down so that we could figure out what information was needed to track progress to achieving “Outstanding Success”.

The next part of the interviews was always the most difficult but in a way even more important.

The next part was to get the interviewee to tell you what the obstacles were for achieving this Vision of Outstanding Success that he/she had just been enthusing about. It is ever so easy to kill the interview at this point in time. The interviewee might become hostile, she/he might become resigned that he/she has tried to get the company to move forward with some of his/her ideas and not yet been successful. Going in to a negative frame of mind happens more often with less senior people I found.

But finding out what the obstacles are and where data might help get around those obstacles is just as important and getting the vision for Outstanding Success defined.

Why? Because any senior manager can give you a Vision for Outstanding Success. That is not the problem. It is execution that is the problem.

I am reminded by a comment I saw one famous Management Consultant make, I can’t remember which one. He wrote words to the effect.

“If I had 10 CEOS from fortune 500 companies in a room and I asked them to write down their vision, mission, goals and strategies for achieving these, and then I mixed them up and handed them back out randomly? Most would have trouble knowing if they got their own back or someone elses.

Why? Because most large companies are in business for the same reason and have the same vision, mission and goals, or very similar. And they have the same strategies to get where they want to go. Those are NOT the things that make the difference in a company. They are necessary and without them you will most likely fail. But they will not make the difference.

The difference is in execution….every, single time.”

I remember reading that in the mid 90s or so and thinking “this so matches this whole idea of Outstanding Success and then searching for the obstacles that are in the way and addressing the obstacles so as to make Outstanding Success more likely.”

And so it has. And it has stood my clients in good stead all these years.

Create the vision for “Outstanding Success”, make sure that all senior managers are well aligned on the vision they created for “Outstanding Success” and then search for all the obstacles that would get in the way.

With those two things in mind? We would gather data into the data warehouse that would assist us on the path to making a reality of the “Vision for Outstanding Success”.

Summary

In this blog entry I have talked about the process that I have used for “Setting the Vision for Outstanding Success” for many clients across a period of nearly 20 years now. I first used this idea on a real project in 1997. Before then we were selling it and talking about it but did not use it end to end on a project.

I talked about how the Interview is an Art Form and not something where you merely take a set of questions and fire them at senior managers like bullets out of a machine gun. I gave an example of how some people can be relied on to react very negatively to the sorts of interviews that I have conducted over the years.

I included some of the questions that I have put to senior managers over a period of more than 2 decades to find out how a data warehouse could make them more money in their business.  Please note not ALL questions were copied to this blog post.

If you develop a technique such that the senior management team of your company define and agree on their “Vision for Outstanding Success” that has a two year timeline and you identify all the obstacles, both internal and external, to achieving that “Vision for Outstanding Success”? You have a very solid road map to go on to develop the data warehouse that will support treading the path to “Outstanding Success”.

Of course, in years gone past we had to select which data we were going to place into the data warehouse because the cost of the data warehouse was driven by the volume of data and the number of data fields. Indeed, by 1997 I was able to give fixed priced quotes for data warehouses based on:

• Number of operational systems.

• Number of files/tables.

• Number of fields.

• Volumes of the biggest tables.

As early as 1997 I could get within 10% of the actual cost for a project based on those numbers alone. As far as I am aware I was the only person in Australia who could do that in those days!

Today, with SeETL and BI4ALL we say “take everything in to the staging area”.

Why? Because the cost of disk/CPU is negligible. And with SeETL the cost of ETL to get data into the development staging area is also negligible. Once all data that MIGHT be useful is in the staging area is it much easier to decide what will go through.

On projects in recent years we have taken ALL data that is not purely operational data into the base layer of BI4ALL.

In the next blog post I will talk about how SeETL and BI4ALL now support the process of supporting the path to the “Vision for Outstanding Success” in a much better way than we were able to in the past, and usually in a much better way than anyone else does it today.

If you got this far? Thank you for reading and/or listening. I hope this was useful to you.

Best Regards

Peter

The post Setting the Vision for Outstanding Success appeared first on Instant BI.

What is Success for a BI Project?

Hello and welcome to this new blog post. Today we are going to talk about one of the most important ingredients of guaranteeing success of a Data Warehousing Project. Namely, being able to define what success looks like so that you have a much better chance of being successful.

If you ask the people on a Data Warehousing Project or “Big Data” project a question like this.

“What does it mean for this project to be successful? How is success defined for this project?”

Almost inevitably you will get answers along the lines of:

• If we make our date that is successful.

• If we have all the data we said we would have that is successful.

• If the users are happy with what we deliver that is successful.

• If we deliver on or under budget that is successful.

Think about that question yourself. On all the data warehousing projects you have been on, or done, or seen, what would people call “being successful”?

The reason success is seen as a function of “budget, timeline, dates, functionality, and happy users” is because this is how IT people perceive “success” for IT projects. I have seen this from day 1 I started in IT. I started in IT in 1982 and “success” was “we go live with this system over the Christmas break”. That is what we were heading for. A three day outage of the plant over the Christmas break and us installing the new computer terminals and installing the new system in those three days. I worked in a steel mill at the time and we were introducing an upgrade of a computer system.

Yes. We did it. We got the system in and running over the Christmas break. And it was called “successful”. It is a VERY rare IT person who looks past these very narrow definitions of success because the USAGE of the system is responsibility of the business users, not IT. So IT does not think in business terms for success.

Now. What is the danger in this approach? It is pretty clear when you look at it. I have seen many data warehousing projects suffer from this danger. The Business People do not have a solid grasp of what they see as successful and the IT people limit their view of “success” to the items mentioned. Budgets, functionality, timelines etc.

When there is this mismatch the likelihood of “success” is usually highly compromised.

Whenever I am asked to review a data warehousing project and give a report on the health of the data warehousing project I always ask to see the Project Charter, Business Requirements and Project Objectives as a minimum. I also ask to see what has been written about the exploitation projects that will follow on from the delivery of the data warehouse.

You might be amazed how often these documents are entirely aimed at a “technology deliverable” and not at a “business deliverable”.

Back in the 90s when we were first promoting the idea of data warehousing our prospective clients had one overwhelming desire as a “requirement” for their data warehousing projects.

“Reduce costs”.

It got so bad that I eventually had this great comeback which was along the lines of

“the best way to cut costs is to close down, is that what you want to do?”

I would persuade my prospects that what they really wanted was sustainable, long term, profit growth so that the company would be solid for many years to come. It would attract quality investors. It would provide good jobs and new job opportunities. It would produce great products and services at competitive prices to win a good portion of clients and retain them over the longer period.

Many of my IT prospects bought in to that idea. And my business prospects bought into it even faster!

In the 90s we were selling the idea of:

“Sure, lets optimise the business to make it run as efficiently as possible, and then lets invest the money we find that is wasted back in to marketing to sell more great products and services to people who need them or want them.”

So when I read these things like Project Charter, Business Requirements, and Project Objectives I can usually tell if the project is going to be a success just from those documents alone. If they are too focussed on budgets,  cost reductions, timelines and dates, features and functions, it is likely that the project will “fail” in that it does not contribute to the profitability of the company.

Now. Over the years I have been able to tell “success stories” in sales cycles without naming the clients. Now that so much time has passed and everyone has moved on from some of my best “success stories” I can repeat these stories without concern for the those involved becoming at all upset about me talking about them. They have long moved on from the competitive advantage they gained from my contributions. These stories are so long ago that there is no one there now who was there then.

But these “success stories” are timeless in that the same issues exist today and the same remedies apply. Further, these success stories will point to how and why my clients have a 100% success rate when they take my advice.
Firstly, what is defined as success on my projects?

Almost always “success” is defined as

“the profit impact of the data warehouse pays for itself in the first year of operation.”

Now. It is true that not every client of mine defined success of my projects in that way. Sometimes we were building infrastructure that was absolutely necessary and if there was a positive financial benefit to it then that was a “bonus”. But in general the “success” we talked about to our clients was “you should be able to point to profit impacts that were impossible without the data warehouse, and they should be more than 100% of the cost of the data warehouse in the first year.”

Now, we said first year, but I was usually aiming at 90 days for break even. By placing the focus on profit we were able to more actively engage the business users in the project and get them focussed on the management decision making cycle and the investments they were making.

So let me give you two sales stories where there was remarkable success where others had tried and failed before us.

Australian Consolidated Press (ACP)

ACP was the largest publisher of magazines in Australia in the 90s. It published the countries two most popular womens magazines, Womens Weekly and Womans Day. It had 37% market share. It was owned by Kerry Packer who was the richest man in the country.

The Magazine publishing industry was highly regulated and it was highly competitive. It was also very mature and was coming under attack from the internet. Revenues and profits were falling and ACP was doing more and more poorly as the second half of the 90s rolled on.

I first tried to sell to ACP when I was working at Hitachi Data Systems in 1997. We lost out to “we will do that internally, no need to hire you expensive consultants.”

I had another crack at ACP when I was working for PriceWaterhouseCoopers. We were privy to some of their internal data and we developed a proposal for a data warehousing program for two years which was the standard sort of offer from PwC. We lost again to “our ERP vendor tells us they can do a data warehouse for us and we are going with our ERP vendor.”

By the time that project had failed I had moved on to Ardent, the makers of DataStage, as the Professional Services Manager. We had to negotiate a tortuous sales process because senior figures in the Packer empire wanted Axciom (part owned by Packer in Australia) to do the work. They did not want to allow the work to go to a third party that had no other relationships with the Packer empire. Axciom were making all sorts of promises and they were a “sister company”. It was a very tough fight but eventually we got the deal because we had the best sales stories of making money for our clients.

By now this had been nearly three years I had been selling to ACP.  Getting a foot into the door of the Packer Empire was something worth working hard on for that length of time.

The timeframe we had to deliver version 1.0? Yep. 90 days. Three years of selling to get a 90 day project.

At the time the fortunes of ACP were poor. Kerry Packer himself called the company “a dog with fleas”. The guy who ran ACP was the best friend of James Packer, Kerrys son, and they had grown up together. Kerry personally intervened and fired the CEO of ACP due to the poor performance of the company. The new CEO, John Alexander, was very dubious about being given ACP as a challenge. Everyone thought that might be something of a poison chalice, and reasonably so. When I met John face to face to talk about the project one of his comments was “I sure hope you know what you are doing because we really do need some help around here.”

Of course, I had NO IDEA what happened inside a publisher because I had never worked on one before. So I had NO IDEA how one might go about improving the fortunes of the countries largest publisher. I was especially worried because it was clear no one at ACP had any idea about how to turn around their fortunes either.

Kerry Packer was the richest man in the country and he was not able to find anyone to turn around this, the largest publishing house in the country? Why would I imagine that I could figure out a problem that had, seemingly, evaded the best publishing minds in the country?

In those days I had a unique way (in Australia) of approaching the Business Requirements workshops that we used to hold. It was technique that I was taught by Metaphor Computer Systems which included envisioning what the client would call “Outstanding Success”. This was a very powerful technique that allowed us to get from the clients what would be called “Outstanding Success” two years in the future if they were able to achieve it.  This would put a “stake in the ground” to aim at in terms of business success.

At one of these workshops one of the publishers made the following comment.  She was asking me, as a consultant to her company, did I know this. She said words close to the effect:

“Do you know that we do not just publish issues of the same magazine? Do you understand that our product is different each week?”

I said
“Well, no, I don’t understand that. Can you please explain that idea to me more?”

She said:
“Sure. It’s like this. We put different people on the cover each week. And we write different articles inside the magazine to suite the audience attracted by the cover. Sure, we cycle through these themes as weekly and monthly magazines. But we might take some months to cycle back to the same type of people.”

And I said:
“That’s interesting, can you give me some examples to help me understand better?”

She said:
“Sure. Lets look at the royals. The royals appeal to a certain set of people. We can put the queen on the cover. We used to put Charles and Diana on the cover a lot. We can put Andrew and Fergie on the cover. You know. Any of the royals on the cover appeals to people who are traditional, who are monarchists , who support the Queen, that sort of thing.

And then we put stories in the magazine about the royals. We put in opinion pieces saying we should stay as part of the commonwealth with the queen as head of state. We get advertising that is royal friendly. The whole magazine will be aimed at people who are royal friendly.”

“And then we have Tom and Nicole. (For those who don’t know Tom Cruise was married to Nicole Kidman and they were an incredibly popular couple). Tom and Nicole represent “new and modern”. They are movie stars, not royals. They have a different base of support. Different fans if you like. Much younger. Mostly less concerned with politics.

So when we put Tom and Nicole on the front we will have opinion pieces about the republic debate, we will talk about the movies that are out now that are suited to young people. It will be a “modern and new” look to the magazine that month.

We do this because if we hit the same sort of demographic month after month we alienate those who are not in that demographic. So even though people who like Tom and Nicole are less likely to buy a “Royals” issue and those who are “Royals” are less likely to buy a Tom and Nicole issue. By presenting different issues with different themes we get more sales than if we pushed just one or the other. Does this make sense?”

And I said:
“Sure. That makes a lot of sense. Your product is so different from week to week that you consider it a new product even if it gets recycled to an extend a few months later. Is that what you are telling me?”

And it was.

This conversation was noted in among all the other conversations that were had. I then had a meeting with the guys who were the experts on the ERP. When I got to this conversation I asked them.

“Please tell me how the number of copies to send to each of the 7,000 locations for each magazine is calculated in the ERP.”

As I suspected the number of copies were calculated based on a straight history of sales to that location. This method left ACP with about 25% of copies unsold of their #1 magazine each week. It also left them with many “sell outs” of their magazines. A “sell out” meaning the retail outlet had no copies left at the end of the magazines life. This is not desirable. You still want a few copies on the shelf so that you keep the shelf space and so not allow a competitor to get that shelf space.

So we added one more idea on the agenda for testing with the data warehouse when we got it. Analysing the data warehouse sales to determine if the cover of the magazine showed correlations to sales in the outlets.

The theory was that if “Royals” vs “Tom and Nicole” truly did appeal to different people then perhaps these people bought their magazines from different outlets because they live in different places. If this was so then the ERP would not be able to know this and it could not adjust the copies allocated to each of the 7,000 retail outlets to match the forecast sales for this cover and this content.

When the data warehouse went live and we had done all the MUST HAVE reports we decided to dive in to this question because sell outs and over-supplies were hugely expensive for ACP. As you might have guessed from telling this story. We found very high levels of correlation between the “theme” of the magazine and the sales at the 7,000 outlets. We got the publishers to tell us all the different “themes” they ran with and we profiled each of the themes.

We then developed a forecasting mechanism for the ERP in the data warehouse and we fed the ERP with our custom forecasting mechanism. Now, we expected some improvement. (I was gone from the project by now so this is what was reported to me later.) But the improvement was MASSIVE. It was not only that we were under supplying outlets where the magazine would be popular. We were over-supplying outlets where the issue would be less popular.

The team were able to learn over a period of about 2 years how to recalculate and recalibrate the forecasting module for number of copies to send to each of the 7,000 outlets based on the profiles and the 10 years of historical sales data we had available. This recalibration reduced the oversupplies. But more importantly, it reduced sell outs as well. So people were no longer buying a competitive magazine so often.

There were other things the team did. This was not the only change. But it was THE BIG CHANGE that led to a massive performance improvement by ACP. Inside 2 years ACP had doubled its gross profit. Market share went from 37% to 42% in a mature, saturated marketplace. Kerry Packer himself called ACP “The Jewel in the Corporate Crown” just 2 years after calling it a “dog with fleas”.

John Alexander, for being willing to take the risks necessary, was rewarded and moved up the Packer Hierarchy, eventually becoming CEO of the group. Kerry Packer himself mandated all his companies have a data warehouse just like ACP. The consultant I put in as the Business Analyst was named as the signatory and responsible for all Packer Company BI Solutions. All Packer companies had to have his blessing on their projects. He called me ever so pleased and said “I have died and gone to consultant heaven!”

That is what I mean by “Successful Data Warehousing Project”. You double the gross profit, you garner 5% market share, and the richest man in the country says that your client has gone from a “dog with fleas” to “the jewel in the corporate crown” in just two short years. After Kerrys death James sold ACP and it was rated as one of the world top 10 publishing houses when sold off.

Just in case you are wondering how long it took to get the investment back? ACP put AUD300,000 in to the version 1.0 data warehouse with later funding to be dependent on success of the project.

So the target was to impact the profit line by AUD300,000 inside the first year.

While we were building the data warehouse the Managing Director of another Packer company came to us looking for ideas. He ran the distribution company. This company delivered ACP magazines to the 7,000 outlets. It also delivered competitive magazines and other products. Their carriers went past all 7,000 outlets at least once a week. Many of the carries went past some outlets 5 days a week Sunday to Friday. The distribution data was inside our ERP which, of course, meant we had access tot competitive information about other magazines which was not allowed to be passed back to ACP publishers.

We started the meeting and the Managing Director of the distribution company explained his problem. They were bidding for the “Readers Digest” distribution deal for the next 5 years. This was the contract to distribute all copies of “Readers Digest” to all 7,000 outlets for the next 5 years. A VERY substantial contract.

He explained that he had been personally informed that his company would not be getting the deal. When informed he begged for one more meeting and he had been granted this concession by the senior management of Readers Digest. He was here looking for ideas. What could he sell to them that he had not sold so far?

We were brainstorming and kicking ideas around. No idea was “too outrageous” because of the size of the deal. He encouraged us to throw even the wildest ideas out there. So the consultant I had placed there says “Why not promise them that our data warehouse will have their sales numbers on their desktops in their offices on a daily basis. 9am in the morning. They will have their numbers.”

He MD looked at him and said: “Can we actually do that?”

The consultant said: “How the hell would I know? Besides, it doesn’t matter if they don’t buy, right?”

The MD said: “I like the guy! Let’s add that to the presentation! It sounds like a great idea!”

About two weeks later the CIO asks the consultant and I to come in to his office. He had a voice message on the phone from the MD of Distribution. They were obviously having a HUGE party. The MD was screaming “We got the deal! We got the deal! We got Readers Digest! You guys have to deliver now!”

Now. I, as a consultant, was never told the value of the deal. But it was well in excess of AUD300,000 that is for sure. Delivering Readers Digest to 7,000 outlets for 5 years would be worth millions of dollars, that is for sure. So, here we were. We still had a few weeks to run on our project even to get it into production and the CIO was shaking our hands saying “Gentlemen, congratulations, I have never seen anything like this in my professional life before. Well done.”

When your data warehouse becomes an essential competitive weapon that wins you multi million dollar deals? You know you are doing something right!

This is what success of a data warehousing project looks like. This is one of my best sales stories because the company in question was in such a mature and competitive industry that was also being pressured by one of the largest technological changes in the last 20 years. The internet. To turn around such a company based on the data inside their ERP and other systems was an amazing feat.

And in case you are wondering? This is not the only example of this sort of thing I have. It is just one of the few that I can so easily talk about since the company has been sold twice since then and no one from that project is still with the company.

Qantas Cargo – Different Techniques Leads to Success

Everyone knows Qantas, the national carrier of Australia. They have an international cargo division that sells cargo space on the Qantas flights and this is a very profitable part of the overall Qantas business. The system that was used to manage cargo was from Unisys and it was not very good. It certainly did not have a data warehouse that would help Qantas Cargo optimise their business.

In the late 90s Qantas Cargo decided they would try and build a data warehouse to optimise the Qantas Cargo business. As I recall it was explained to me that the prior three projects had failed. The new project manager had been told that if this effort failed then he should resign as there was no way forward for him in the company. The man, lets call him Fred, had been in airlines for 20 years so he was very concerned that this project would be a success. He later told me he called 10 or so of his most trusted colleagues and asked them “If I have a really tough data warehouse project that HAS to work who should I hire?”

Fred told me much later that 6 of the 10 or so gave my name and no one individual got more than one mention. Fred had decided that with that sort of reputation I was the man for the job. Fred still ran us through the hoops of a “tender” to squeeze us on price etc. But we were, apparently, destined to get the job as long as I presented as his colleagues had claimed I would.

By this time I had been “Constructively Demoted” back to a consulting role and I decided that Qantas Cargo was so important to us that I would to lead the project myself. Fred was right. The Unisys system was a nightmare and this would be a very tough project. It was no surprise to me the three previous efforts had failed. The reason for the importance was that Qantas had bought in to an Enterprise License deal for DataStage and this would be the first large DataStage project at Qantas. So the future of DataStage also hinged on this project.

As I said above we ran a unique workshop process. However at Qantas, for reasons that I can not recall, we decided to run the process as a series of interviews. So I set about interviewing the 20 most senior people at Qantas Cargo. The schedule was 3 or 4 interviews a day. No one can do more than 4 interviews a day at that sort of level.

We also did not record these interviews as there was sensitive information to discuss about personnel. It is often my habit to record interviews on tape. But when personnel issues are being discussed recording will stifle discussion because no man will ever believe that a recording can never find its way in to the public whereas “conversation” can easily be dismissed as “a misunderstanding”. When you do not record interviews the work of the interview is much harder because you have no backup tape to go to. I just wanted to set that as the scene.

So I ran our “Outstanding Success” technique that is so incredibly successful for all the clients that we use it on. But it is not what people expect. And we ALWAYS get complaints about it.

Qantas Cargo was no different. About 2 weeks in to the project Fred and I were summoned to the CEOs office. The conversation, summarised, went like this.

CEO:
“My people are complaining that you do not know what you are doing? Can you tell me what this is all about?”

Peter:
“Sure. I bet they are saying things like “He is not asking us the question we are expecting. He is not asking us what data we use, what reports we use, what reports we want, those sorts of things.” Would I be correct?”

CEO:
“That’s quite remarkable because that is very much the flavour of what they are asking. They tell me you know nothing about Cargo, about our business, and you are not asking them questions that would lead you to know about our business and our reporting needs. Quite frankly, they don’t trust you or believe you. You have a real problem here.”

Peter:
“So may I ask you a question?”

CEO:
“Sure”.

Peter:
“The project has failed three times before we were asked to come in. I have the documents from the failures. Those documents define the data your managers say they need. They define the reports they say they need. The interviews run and the documentation produced is exactly what your managers think I should be asking about, correct?”

CEO:
“Correct”

Peter:
“Have you ever head the saying that the definition of insanity is to do the same things over again expecting a different result? Your projects failed precisely because people asked those questions. Those sets of questions are not sufficient, or even necessary, to make a data warehouse project successful. We take a different approach because we developed it over a long time and we improved it and it works. We improve it each time over.

Fred here hired us because we are the best in the country. And the reason we are the best in the country is precisely because we do not do what anyone else does. We do things differently. And because we do things differently we can guarantee the success of your project if you do as we advise you to do. You will see this when we show you the prototype in 6 weeks time.

So. Will you take my word that the reason we are not doing what your senior staff expect, what they have done three times before, is because we know that does not work? Will you take my word that we are doing things differently because that is how we guarantee success of our clients projects?”

CEO:
“Very well put young man. I look forward to the prototype demonstration.”

He shook my hand and I went on my way. Fred was VERY impressed at the meeting. We laboured long and hard on the prototype.

When it came the day of the prototype I did the talking and one of our team ran the demonstration.

The result was stunning. I have to say the team had done a remarkable job in just the 8 weeks we had to do the requirements and build the prototype to ask for the money for the build. Every team member deserves the greatest of praise for their dedication, commitment, hard work and just plain brilliance at their jobs. It was a pleasure to present the prototype. It was one of the best I had ever been involved in building.

Those who had been “complaining” just 6 weeks earlier were amazed at what they saw presented. They were making comments like:

“It is like they are mind readers, this is what I need, but I never once said this is what I need. How the hell do these guys know what we need when we didn’t even tell them?”

Eventually, over all the excitement, the CEO raised his hand and hushed the very excited crowd.

CEO:
“I have to say, that is one of the most remarkable demonstrations I have ever seen in my life. You told me 6 weeks ago you have never worked in Cargo, is that true?”

Peter:
“Yes. I have never worked in cargo. I have not even worked in anything like it before. Not ever.”

CEO:
“The presentation you just gave demonstrates a level of understanding of our business that I would be impressed by if someone who had been here 10 years gave it. You understand our business better than many people who have been here for a decade. I have to ask. How did you do that?”

Peter:
“Sure. You will remember 6 weeks ago you invited me to your office to discuss the complaints of the people in this room that I was not doing what they expected.”

CEO:
“Yes, I remember.”

Peter:
“You will remember that I told you we do not ask the same questions as everyone else. You will remember I said that we have a unique technique and a unique way we go about our job and it is because of this we get different results. You will remember I told you that we are the best there is at this?”

CEO:
“Yes, I remember.”

Peter:
“Well? This is what that looks like. The way we approach our projects allows us to develop a very deep understanding of the drivers of our clients business very quickly which means we can then build the charts, graphs and reports we know you need to generate outstanding success over the next 2 years. In short, we can figure out what you need better than you can figure our what you need because we use a different approach.”

CEO:
“Ok. I have to admit. This is impressive. I think I speak for all my senior staff when I say what we have seen today as a prototype? This is what we need.”

Peter:
“Great. Now there is this issue of money for the project. I am sure Fred has some paperwork for you and your staff to sign off on to approve the budget. May I presume that is all ok with you? That we can approve the budget today?”

CEO:
“Yep. You have your money. Come back on Monday and get this thing to production as fast as you can.”

Now…there were plenty of bumps in the road ahead for that project. Not that we need to go in to them in this blog entry. But by the end of the next year the CEO declared the project was “the most successful project ever run in Qantas Cargo”.

I am not personally privy to the financial aspects of the results produced as, again, I had moved on. It is sufficient to say that the project team I left behind remained working there for about 5 years.

Our technique of “Envision Outstanding Success” two years into the future was integral to making Qantas Cargo such an outstanding success of a project. The business managers were absolutely signed up and committed to making those visions come true. All we really did was give them the graphs, charts, and reports, they would need to look at on a daily, weekly, monthly basis to track the key business drivers that would indicate they were on the right path, or, indeed, deviating from that path.

The team later built all sorts of applications to detect how to optimise various areas of the business processes and optimise use of space/weight allocations inside the aircraft. But it was the “Envision Outstanding Success” two years into the future technique that allowed the senior managers to clearly and accurately put a stake in the ground as to where they wanted their department to be in 2 years time that allowed us to build one of the best prototypes I have ever done.

Of course, in those days we had no “prepackaged” models and so we still had to run these Business Requirements interviews before we built the prototypes. We now delay the whole “Business Requirements” process until we have analysed the data that is available so we know what is possible and not possible as we interview. This has been a rather radical change in our procedures which has been enabled by SeETL.

In Summary

Too often “success” in a data warehousing project is linked to budget, features, functions, timelines, delivery dates.

This takes the focus off what should be the REAL measure of success. Profit contribution. More specifically, “can we get a financial payback for the data warehouse in the first 12 months of operation?”

By focusing in return on investment, by focusing on achieving what can be agreed as “outstanding success”, you look beyond the end of the data warehouse build and look into the realm of the exploitation projects that will be run out of the data warehouse after its implementation. That keeps the business people far more active and far more engaged than they might otherwise be. In short, it keeps their attention. It gets them ready to use the data warehouse for exploitation when it comes into production.

Where this connection to the exploitation projects is weak or lacking it is that much harder to actually make the business process changes or run the business projects that will actually make the money you need to make from the data warehouse to pay for it.

I used two old but still very relevant examples of how I have done this in the past to deliver massive value to the clients that were on the receiving end.

If you got this far? Thank you for reading and/or listening. I hope this was useful to you.

Best Regards

Peter

The post What is Success for a BI Project? appeared first on Instant BI.

Share

Original Post

4 Key Questions About Your BI Methodology

Hello and welcome to this new blog post. Today we are going to talk about one of the most important ingredients of guaranteeing success of a Data Warehousing Project. This is a “do not pass go without it” situation.

I mean the methodology that is going to be used during the project. I see so many projects that have failed and when I ask “what methodology were you using”? the answer very often is “Agile” or sometimes even “no specific methodology” at all.

Let me say this right here, right now. If you are not using a data warehousing specific methodology then your chances of success are very significantly compromised. It is not that a methodology will guarantee you success. Far from it. But NOT using one is asking for trouble.

I have a massive amount of experience with IT development methodologies in general and BI methodologies specifically. The first methodology for BI that I was exposed to was the Metaphor Computer System Methodology in 1993. I wrote the BI Methodology for SAS Australia in 1995. I wrote the Hi-STAR Warehouse Methodology for Hitachi Data Systems in 1996. I used the Price Waterhouse Coopers Methodology in 1998. I used the Prism Data Warehousing Methodology in 99-00. I used the Sybase Systems Integration Methodology from 2001-05.

I do not know any other person who has had exposure and used as many of the leading data warehousing methodologies as I have. If there is someone out there I would be interested to meet you! So I know data warehousing methodologies. I know good from bad.

We have developed our own methodology that we have adapted to be tightly linked to our products. Our methodology is a result of the many years of work that I have been involved in over the years. We have produced a one hour video about our methodology. You can download the presentation slides from this link.

Naturally, I am not going to repeat our methodology presentation in this blog post. You can go and hear what we have to offer when ever you would like.

In this blog post I am going to give you four key questions to ask about the methodology you are using. If your vendor can not answer these questions off the top of their head sensibly? You should start asking some harder questions.

1. Is it a waterfall methodology?

This slide is the most important slide in our whole sales kit. It shows the way in which our software enables us to alter the way we deliver projects. Click on the image to get the full sized image.

The traditional approach to building a data warehouse is to follow a waterfall methodology of some sort. The phases run something similar to:

• Requirements

• Data Models Design

• Data Mapping

• ETL Design and Build

• Applications Building

• Testing

• Move to Production

Once the applications building has begun and new ideas are forthcoming about what would be valuable those ideas go in to 2.0 which may or may not ever be implemented.

Typically these phases have some level of overlap. The greater the overlap the greater the risk of the project being unsuccessful becomes. In an ideal world where you have unlimited time and money you would not overlap phases to reduce the risk. But we do not live in such an ideal world. Not by a long shot.

The biggest problem with this style of data warehouse implementation is that you do not get to start building applications and you do not get a populated data model until after you have coded a significant portion of the ETL in your ETL tool.

We have always tried to build prototypes to reduce the risk. But in the 90s, when we were dealing with complex ETL and large volumes, we could not build prototypes in anything more robust than Microsoft Access. These were tiny and simple prototypes that very often had significant flaws in them. We did our best but often our best was fraught with problems. Those were tough times to get the data models as good as they could be before sitting down and coding the ETL in a vendor ETL tool like we used to do when I was the Professional Services Manager at Ardent.

When I wrote my list of features that an ETL tool needed they included:

1. Typeless.

2. Mappingless.

By this I meant that the ETL tool itself should have no idea of data types so that there can not be a problem with data type mismatches. Data type mismatches were the bane of our ETL developers at Ardent. The slightest mismatch would generate a message that we would have to go and fix.

I also meant that the ETL tool itself should not know what mappings exist. The ETL tool should discover the mappings that are needed at run time. I proposed that there should be no dictionary where the mappings were defined or where the source and target tables should be defined.

I proposed that the ETL engine should be told which source and which target it was to use this time and that it should open the source, open the target, discover the column names, build a map of movements according to column names, and then process the data based on column names.

I proposed that the dictionary should be minimalist and that most data should be stored in the database catalog itself and read using ODBC to give greater transportability.

By doing this the processing could be abstracted to one program per type of processing and that one program could perform all the processing for that type of processing for all mappings.

Naturally, the ideal place to store the sources and the targets was inside the database as a view and not in some external catalog that would be used to define the processing.

This was implemented in the SeETL C++ engine and it was very successful.

This had the added benefit that it became possible to implement the prototypes to full scale before migrating the SeETL ETL to the vendor tool of choice.

Because SeETL enabled us to build the prototype to full scale we were able to collapse the “waterfall” methodology so that model design, data mapping and ETL design and build all happen simultaneously. This collapses the time line of the project, drastically increases productivity, and reduces risk.

Further, because the applications can be built starting much earlier feedback from the applications build can be fed back in to the data models design process.

If the client insists on using an already installed vendor ETL tool then the SeETL to ETL tool migration happens right at the very END of the project. And we have tools and methods of doing that very quickly and easily with good QA processes.

So the very first question you should ask of your methodology is this “Is it a waterfall methodology”? And if it is? That is a major contributor to risk. It is not appropriate and it is unnecessary.

You can compare the diagram we use with what you are doing to determine how you compare.

2. Do you have to spend major dollars before you are sure of what you will get?

I have bid for and won the largest data warehousing tender of the year in my country twice. Once for the Australian Customs Service in 1997 and a second time in 1998 for Telstra, the ex government telco in Australia. I have also won many millions of dollars of BI business in other tender responses.

One thing that I never particularly agreed with in the tender process is how the customer has to make a decision and back that up with a major amount of money BEFORE they know what it is they are getting. It is ok when you are selling hardware or software where you can read the specifications or manuals and you can be sure of what you are getting. You can sue or not pay if the delivered product does not match the manual or the specs.

But building a data warehouse is very different. As Bill Inmon famously said in the early 90s

“You don’t really know what you want from your data warehouse until after it is built and you start using it.”

This absolutely reflected my experience as well in the early 90s. My very first data warehouse “Business Requirement” was “I want to be able to ask any question I think of. I do not know what those questions will be. And I want the answer before I forget why the hell I asked the question in the first place.”

I mean? How do you tender for a system to do something like that?

You can’t.

Too often companies buy their data warehouse via a tender process where they have to make a decision and a major financial commitment before they know what it is they are going to get for their money. Sure. They have “business processes” to go through but it is a business process that increases the risk of failure far more than it reduces the risk of failure.

Tendering for projects is supposed to reduce risk and reduce costs by getting the vendors to bang their heads against each other over features, functions and prices. In areas where the delivery is well defined I agree. It does do this. You want to buy hardware? You want to buy software? Put it to tender and get the vendors to beat each other up on price. Great!

But to build your data warehouse? You should get the “almost finished” product before you put down the big dollars, pounds, euros etc.

If your vendor is wanting to lock you in to spending the big dollars and making a large financial commitment before you know what you are going to get? This should ring alarm bells very, very loudly in your ears.

Specifically. You should be able to have a reasonable sized prototype built before you put your money down for your production hardware or software. And that includes ETL tools which are quite pricey.

This all comes back to the ability to prototype your data warehouse before you have to put the big money down and that means prototype ETL or ETL that is free until you are mostly finished.

We make SeETL free for those people who sign a contract saying they will not deploy the ETL into production. It is our way of giving the “try and buy” option.

If our clients also use BI4ALL they can inspect the data models to a great extent before buying. For example? We have 7 hours of video that takes our prospects through the data models end to end so that they can understand what it is they are buying before they have to spend any money.

Further? We now license the BI4ALL models on an annual basis. If you do not like what you are getting then you don’t pay license fee and you decommission the data warehouse in favour of what ever else you want to implement.

But the fact remains. If your vendor is driving you to spend the big dollars before you know what you are getting? That should ring alarm bells. Your vendor should explain to you how they are free to stop at any time until the prototype is signed off as being what is desired and needs to be productionised in the chosen production tools.

Of course, for your part you must understand that prototypes are not production ready systems and you need to have the patience to give your IT people the time to turn prototypes into production systems. Too often I see business people put pressure on IT people to productionise a prototype rather than to make sure the job is done properly.

3. What is the level of transparency of progress?

It never ceases to amaze me how vendors can get away with the lack of transparency of their project work that they do. Really. As a vendor in many of my projects I went out of my way to make sure that progress was as transparent as possible, warts and all. This makes for more successful projects. Hiding problems from your client might be good in the short term but it is bad for business in the longer term.

To give you an example of the level of transparency you should be asking for, if not demanding, this is how we make transparency possible.

The SeETL workbooks are ALL able to be loaded into our dictionary. We are the only people who have an end to end BI development dictionary. Where the product does not make its metadata readily available, like the reporting tools, we have proxy documentation that allows us to enter metadata about the tools in to our dictionary. We do the same for stored procedures.

As the project progresses spreadsheets are versioned and loaded so that old versions can be queried. This means there is no temptation to be “fiddling the numbers” from week to week in reporting cycles because last weeks data is still in the dictionary. Fiddling the numbers will be exposed quickly and easily so it will not be done.

The dictionary is open and easy to query. We provide SQL Server Report Services Reports for all the information in the dictionary. If you would like another query tool to report on the dictionary you can re-create the reports using any tool you would like.

Our tools record everything from the Business Requirements and Project Objectives all the way through to every field in the data warehouse, every field on every report, and every piece of ETL.

During the development of the ETL process we track the progress of each mapping through the development process.

Creating the weekly “Progress/Status Report” is little more than running reports off our dictionary and presenting them to the project management group that reviews such projects. There is simply no way for a project manager to hide problems when our methodology is married up to our tools.

We can give counts of fields mapped, percentage progress, level of testing that the mappings have been put through. Everything. Our clients find our level of transparency better than any of the other vendors they deal with.

We even have project planning worksheets if you wish to use those instead of a normal project management tool.

If your vendor solution does not have full transparency in reporting? If you have to “take the vendors word for progress”? That should be ringing alarm bells for you.

4. How does the methodology reduce communication paths?

Software development is all about communication paths.

How many are there?

How is information communicated along those communication paths?

The more communication paths you have and the less fixed the format of the communication the higher the risk. The higher the chance of failure. Communication paths have been the study of software development since Frederick P. Brooks wrote “The Mythical Man Month” in the 70s.

Again. Let me explain how we address this issue so that you can compare your situation.

Our workbooks allow the collapse of the tasks for data modelling, data mapping, and ETL design and build to all occur at the same time in the same tool. An excel spreadsheet.

There is no separate exercise of designing a data model and putting it into a data modelling tool.

There is no separate exercise of documenting source to target mappings in word or excel and then giving those specifications to a programmer to code them into an ETL tool.

The data modeller, ETL designer, and ETL coder are one and the same person. There is no communication path needed to anyone else. This rather remarkable situation is dependent solely on the ability of the workbook to capture all that information. We actually recommend that all data modelling, ETL design, and ETL implementation for the prototype be done by the one person per project. Of course, this is a very high skilled job because of the diversity of skills needed. This was normally my job on projects over the last 12 years.

On some projects clients like to do some of that work to learn. Our general approach to skills transfer is to get the DBA for the project to be the reviewer of all this work. It is the DBA who has to take the workbooks and physically implement the database. But this is only a part time job. So the DBA is used to get answers to questions for the designer so that the designer does not need to chase answers to questions.

The mechanism for “communication” is not imprecise text in a word document. It is source code in a workbook that can be used by a generator. This is a much more precise and reliable way of communicating what needs to be communicated.

So compare this to your project. If you have one person designing the database, another designing ETL, another group coding ETL? Then you have numerous communication paths, all of which can have failures along the way.

The reduction of communication paths and the preciseness of communication by using coded workbooks vastly reduces time, effort and risk when designing and building a data warehouse.

This is true whether you use a packaged model or whether you are going to design a custom model for your company.

One of our clients insisted that they wanted everything that needed to be remembered in the workbooks and in the dictionary so that as staff turned over they knew for 100% certain that everything they needed to know about was documented in the workbooks.

Compare and contrast this to your situation. Do you have different pieces of code in different places and different tools so that it is possible to “forget” some component when you are attempting changes?

That is quite normal in companies that do not use our tools. We see it all the time.

Summary

In this short blog entry I have presented 4 very specific questions you should be asking about the methodology you are using to implement and/or maintain your data warehouses.

They are again:

1. Is it a waterfall methodology?
2. Do you have to spend major dollars before you are sure of what you will get?
3. What is the level of transparency of progress?
4. How does the methodology reduce communication paths?

These are, in my mind, the 4 most important questions you need to be asking about the methodology that you are using.

If your BI vendor/consultants can not present you with a sensible answer to these question you should be hearing alarm bells.

The methodology you use to deliver your data warehouse project will be one of the most important factors towards your success. It will not guarantee success but it will go a long way towards that goal in and of itself.

And, of course, a bad methodology will increase your risk.

If you want me to review the methodology and approach that your vendor is using? I would be more than willing to provide you a written report for a reasonable consulting fee.

If you got this far? Thank you for reading and/or listening. I hope this was useful to you.

Best Regards

Peter