Big Data: And the hype played on

MARTYN RICHARD JONES

In spite of the best-efforts of Hadoop evangelists, consulting houses, and IT infrastructure and service vendors, Big Data – hailed as the greatest thing since the dawn of greatest things – is failing, and dramatically so, to produce the necessarily corresponding quantity and quality of tangible, detailed and verifiable success-stories.

So, given the dearth of Big Data success stories, why are so many in the industry still banging on with troll-like insistence about the ‘indisputable’ relevance, importance and universal applicability of Big Data?

Seen from where I stand, and I don’t think I am particularly unique in this respect, a lot of arrows are pointing towards a quieter future for Big Data, one where it has far more limited use than was once imagined, a data niche rather than a data driven all-encompassing market-based revolution. But obviously not everyone is sharing that view, at least not ostensibly so.

The fact of the matter is this. That in spite of what the good people at the prestigious house of Gartner have been saying, the Big Data hype-cycle is far from over. In fact, we have entered what might be crudely yet accurately described as the golden age of Big Data boloney.

When I survey the Big Data landscape and see companies, organisations and individuals continue to bang on the Big Data drum like as if it was going out of fashion, it’s quite embarrassing. Acutely so. Even for someone who isn’t participating, either actively or passively, in the Big Data dog and pony bestiality-show.

So, what’s happening with the Big Data in-crowd? Words were coming to my mind, words such as overcompensation, projection, doublethink, denial, delusion, social, awkward, wacky correlation and non-existent causation. But none of these were entirely satisfactory. Lacking, as they do, in conveying any sort of coherent, tangible and credible explanation of the continuing and burgeoning Big Data hype phenomena.

Then it dawned on me. Black tulips.

Industry players and Big Data pundits are inexplicably engaged in what is at best a zero-sum game, and at its worst has clear parallels with the hubris, wilful ignorance and  greed which lead to the last major global financial crisis and also has nasty parallels with the dot-com bubble.

So, the next time you are reading a gushing puff-piece on Big Data take this advice to heart. Instead of thinking about the author as a friendly adviser, think of them as an unqualified, unregistered and unregulated financial adviser who is trying to flog you something that they probably don’t understand and that they have no idea how to value – remember, all they are thinking is in their ‘commission’.

More to the point, don’t go repeating the same talking points as if they were facts you can take to the bank, when you yourself are unsure about how reliable those ‘facts’ might be.

Stated simply. There is very little which could be more suspect than an announced global revolution in Big Data, a revolution that is accompanied by a dearth of tangible, detailed and verifiable success-stories. So take good note of what is really going on.

If someone deceives you once, it’s their fault, if they deceive you twice it’s your fault. If you then go on to ‘do unto others what has been done unto you’, then expect to be rightfully derided and decried.

Be professional, be on the side of the angels, and do the right thing. In the longer run you will thank yourself for your wise choices.

Many thanks for reading.

Something funny happened on the way to the Data Lake

When one surveys the vast array of Big Data tomes and the even vaster output of the regularly amazing Big Data bullshit babblers, one thing that strikes you immediately is that Big Data isn’t explained starting with a story about the marvellous useful applications that you might imagine are generating all of this additional Big Data, but instead they almost invariably dwell on all of the data generated by social media applications of dubious business value and even more dubious social value.

Basically we are being led to believe that applications of peripheral interest, that is, peripheral in the grander scheme of things, are capable of simultaneously meeting almost all of our immediate needs and also of generating a voluminous wealth of data detritus that is somehow going to bring about an amazing economic revolution, the awesome galvanising of innovation and the creation of absolutely fabulous riches beyond the dreams of avarice.

What’s more, every day some Big Data proselyting dope has a cunning plan and pops up on the web like a drunken gopher with some extrapolating and theorising boloney about the quite unusual successes of very unusual businesses such as Facebook, Twitter and Google and then hammers square pegs into round holes in order to produce a generalised theory of awesome Big Data irksome-foolishness that will certainly mean, without a doubt, that we will see the raising of the luck and fortunes of all businesses, everywhere, if only they would buy into the Big Data religion, now.

If you don’t believe me then check it out for yourself. For example, take a look at the Big Data channel over on LinkedIn, now that it’s been apparently taken over by the master guru of astro-turfing Big Data bullshit and babble himself, it now comes with wall to wall Big Data hype, and it’s all the same, and it’s all so obviously wrong that it makes you wonder if LinkedIn is not deliberately converting itself into some species of professional laughing academy, for idiots by idiots, with no room for even the occasionally and superficially contrarian.

In fact, looking at some of the latest blunderbuss blasts of blatant bullshit I am reminded of the quote ascribed to Groucho Marx which I will makeover for modern-day consumption: “These are my Big Data principles, and if you don't like them... well, I have others.”

Big Data with BIG SMILES

Having got your attention I would like to introduce you to a pragmatic, real-world and business centric approach to Big Data and Big Data Analytics. When I say that this is the best approach to Big Data you are ever likely to find in the whole universe and in your entire life, I am still significantly understating the magnificent utility, timeliness and the here-and-now facets of the approach.

Now with the introduction done and dusted with, and the virtues of the BIG SMILES approach exalted, it should come as no surprise that this eminently sensible, highly rational and thoroughly reasonable methodical and no-nonsense technique has been applied successfully in more than 500 business-oriented situations.

Best of all, this amazing Big Data approach is free of charge and with no-strings attached – you don’t even have to buy my book. Now, isn’t that amazing?

Let’s start with the basics. The BIG in BIG SMILES refers to Business Insight Gains. This refers to the focus of SMILES. Simples, right? Now what does SMILES refer to:

Fig. The process chain of Big Data SMILES

SMILES is also an acronym, and it refers to the six major components of the SMILES Big Data approach (as illustrated above). Or, more precisely the various phases of the approach. Namely:

1. Start with a significant data-centric business challenge
2. Model high-level options and approaches
3. Implement your chosen option
4. Leverage the products
5. Evaluate performance and value
6. Socialise the outcomes

Let’s take a look at each of those aspects of Big Data SMILES in a little more detail.

1.      Start with a significant data-centric business challenge

It makes sound business sense to start any business initiative with a compelling business reason. If you don’t have one then don’t start, it’s as easy as that.

Fig. Start SMILES

Now, having identified your significant business challenge you should ask the following questions:

• What: What do you want to accomplish with respect to the significant business challenge?
• Why: Why do we want to address the challenge?
• Who: Who should be involved in helping address this challenge?
• When and where: Can you identify the time and place the challenge first comes into effect?
• Windows of opportunity: During what periods can we most effectively address the challenge?
• Which: Can you enumerate the requirements and constraints associated with the challenge and the possible responses?

When compiling your view of the significant business challenge, you could look for example at questions along the lines of:

1. How do I find the Big Data I need?
2. What is the original source of the Big Data?
3. How was this summarization, enrichment or derivation created in the Big Data?
4. What queries and mechanisms are available to access the Big Data?
5. How have Big Data related business definitions and terms changed?
6. How do interpretations of the Big Data/Data vary across organizations?
7. What business assumptions have been made that are related to this Big Data?

Don’t forget, asking great questions about significant business challenges will lead to even more questions, which is where you will want to highlight the new questions that could quite possibly be answered, wholly or partially, using Big Data. Also, don´t confuse great questions with complex questions, the idea is not to impress the audience but to identify and address the challenge.

Another important thing to point out in relation to the SMILES approach is that you should never, ever, in no way shape or form, try and boil the ocean. That is, do not try and implement and leverage Big Data analytics in your organisation in big leaps and bounds. Start out with baby steps, and treat it like a game of tennis. Win points, games, and sets and beat challenges and bad decisions one-step at a time.

Finally, make sure each iteration of SMILES starts with an objective that is big enough to be significant yet small enough to be doable in a reasonably short-time scale, one preferably made up of sprints of no more than 5 to 10 days.

2.      Model high-level options and approaches

Here we are looking at a process of discover and insight; conceptualisation and creativity; deign and innovation; and, prototyping expertise and domain capabilities.

Fig. Model SMILES

This phase has three parts:

1. Defining the problem and developing options
2. Evaluating and selecting the best model
3. Finalising and developing the implementable prototype

Defining the problem and developing options

Using Co-operative Prototype Ideation, Concretisation and Realisation (a unique rapid development feature of SMILES), you may now choose to develop options and models to various stages of maturity and extensiveness.

Evaluating and selecting the best model

Through a cycle of hypothesise and test you may arrive at the model most suited to your needs. If however, no such model is forthcoming or the best model simply is not good enough or promising enough then do not proceed to the next step. Make sure you have a good reason to progress and inertia is definitely not a good a reason, and neither is ‘because we have to do something’.

Developing the implementable prototype

In this phase, you develop the chosen model through to the stage where it is ready ‘productised’ in the following prototype implementation phase.

In addition, as part of this phase, you will be looking at which technology options might provide the best fit with your requirements. Common technological options are typically associated in one way or another with the Hadoop ecosphere, so your problem may be adequately addressed using technology such as Hive, Pig, Bash, Spark or Python, or indeed with products such as MapR, Neo4j or EXASol.

3.      Implement your chosen option

In this phase, you take your well thought out proof of concept prototype and turn it into a business ready and production hardened product.

What is involved in turning a Big Data solution prototype into a product?

Here we are focusing on tools, build competence and teamwork; product piloting and development support; and, the realities of production and support.

Also, as a closing message for this phase description, please note that the implementation phase also includes the active participation of the development sponsors and target user group, as should every phase up to this point, and all of the follow on phases.

4.      Leverage the product

It’s been designed, prototyped, built and productised. Now what?

Well, here comes the moment of truth.

In this phase SMILES can help Big Data teams quickly react to changing business and market needs by capturing and managing new and changing requirements, and by constantly monitoring feedback. The framework can be totally integrated into true agile development processes especially where requirements are extremely dynamic and free-flowing but nonetheless must be managed at a complete Big Data product level.

5.      Evaluate performance and value

An assessment of the value of the Big Data initiative should be made periodically. However, there should be at least four event-pegged must-do valuations carried out during the life-time of the project products.

Fig. Evaluate SMILES

1. Initial acceptance criteria alignment and qualitative valuation.
2. Maturity performance and tangible ROI contribution. Include both the mitigation and avoidance of loss and the enablement of all direct and indirect gains.
3. Life-time-value-to-date to be carried out before all major enhancements.
4. Sunset life-time assessment and valuation. On replacement or withdrawal from service.

6.      Socialise the outcomes

These are the activities that generally put the smiles in SMILES.

Whatever happens with your initiative you must never fail to socialise the outcomes, for as kitsch, cute or painful the exercise may appear to you or anyone else.

Put it this way. You’ve gone to all the effort and trouble of making a Big Data initiative work, and it’s working well, people like it and it’s delivering value. So, what else is there to do? It’s a success, so you shout it from the rooftops, tell all your colleagues and peers, put the news on the intranet and in the company house magazine, and organise a party.

If your project is killed off early or late, or simply fails to deliver value, and sometimes this just happens, then hold a wake, in the Celtic manner, and learn all the lessons that are worth shaking a stick at and make them part of your corporate data management story and knowledge base.

The golden rules of SMILES for beginners

When entering the new dynamic world of Big Data and Big Data Analytics and to ensure that SMILES delivers the kind of success that others enjoy, you must take ensure that the 9 golden rules of SMILES are also followed. These are the golden rules for beginners.

Fig. The 9 golden rules of SMILES

1. Infrastructure – Ensure that the infrastructure is adequate for the needs of the project and ensure that executive management disconnects your prototyping, development and deployment cycles from the necessarily rigorous, time-consuming and constricting requirements imposed on core business operational systems. Remember this is operational, but it is not life threatening, customers will not be lost nor will serious money be burned. So make sure your senior management unchains Big Data from Big IT bureaucracy – and if possible, on a forever basis.
2. Pilot – When you first take on Big Data and Big Data analytics, always start with pilot prototypes and projects. Again, small enough to be doable (avoid overreach at all costs) and large enough to be significant (small and useful doesn’t have to be trivial).
3. Timescale – Aim to deliver initial pilot Big Data prototypes in around the 3 to 6 week mark, and aim to get the first projects into the leverage phase in around 3 to 5 months, tops.
4. Long-term – Aim to deliver fast, simple and elegantly, but also keep a keen eye on the long-term prospects and issues for Big Data and Big Data Analytics.
5. Cash-flow – Control the cash but make sure you have enough to do what you need to do. Focus on value, keep sprints and iterations short, and be intelligent in the management of funding. (see comments on funding later in this piece)
6. Continuous involvement and justification – Justify every decision in terms of business (mandatory) and technological (optional) drivers. Involve business partners, continuously. Make this an absolute mandatory condition for starting the project and continuing. If involvement from business stops, then stop the project until the implication and involvement of the business stakeholders picks up again. Make all of this clear from the outset. Continually seek and reaffirm business justifications for the projects existence – this is a showcase, and people are watching.
7. Sponsors – Ensure that your project has high-level business sponsorship. This cannot come from IT and it cannot come from the CIO or the CDO, unless your Big Data project is to measure and report the performance of aspects of IT and Data Governance.
8. Clean – Make sure that the data that you use is to the quality levels required. The data that you analyse must be at least as good at that stage as when it was sourced. In many cases you will need to scrub and clean data, especially when it’s coming from badly designed, tragically engineered and shoddily built web applications where the designers and developers have only had a passing acquaintance with sound database engineering principles, if at all.
9. Tenacity – finally, never give up until it’s time to do so. If you believe that success is achievable then go for it. If you see that the project is on a suicide mission then kill it quickly, don’t wait until you’ve burned all your hours and cash.

When using SMILES keep these 9 golden nuggets of rules in mind, and you won’t go far wrong.

A note on funding

Try to make sure that your funding aligns with the phases of SMILES.

However, split your funding requests into three parts.

• Start with a significant data-centric business challenge. 2. Start with a significant data-centric business challenge. 3. Model high-level options and approaches.
• Implement your chosen option.
• Leverage the products. 6. Evaluate performance and value. 7. Socialise the outcomes.

This ensures an optimum allocation of resources and provides additional executive and project management safeguards and options, and helps to ensure alignment of contractual assurances and obligations with committed and planned budgets.

A note on testing

Testing is an integral element of every phase of the SMILES approach. For brevity the approach has not been detailed in this document, but the philosophy is essentially one of test early and test often. Under normal circumstances the SMILES approach does not require a User Acceptance Testing phase, and if we are in shops that do require this testing then the UAT becomes a mere bureaucratic formality.

Things to remember

Some people may be surprised that the Big Data SMILES approach does not start with a strategy. In my view, the idea that strategy can begin without the need for identifying a significant challenge is to get things wrong on two counts: i. It’s not the way to go about strategy, and ii. It’s not the way to go about Big Data.

The BIG SMILES approach is not in itself a strategy, it is a guide, reference and framework for those who wish to develop a specific Big Data oriented strategy for addressing a significant business challenge. This is where it is powerful, useful and relevant. It’s a roadmap, a cookbook and a method to understand issues, formulate questions, and provide adequate, appropriate and timely responses, whilst separating the wheat from the chaff, the core from the periphery, and the important from the inconsequential.

Lastly, if you or your Big Data analysis, design development team haven’t done so already, I suggest that you take a look at the Cambriano Information Supply Framework, which provides a solid-basis upon which to architect and design Big Data oriented solutions.

That’s all from me for now. Have fun and enjoy the Big Data journey.

Thank you for reading.

If you would like to know more about SMILES, the Information Supply Framework, Core Statistics, Core Data Sourcing, Data Governors, the Analytics Data Store or 4^th generation Enterprise Data Warehousing then please drop me an email or visit:

http://www.cambriano.es

On a lighter note, readers may also be interested in joining The Big Data Contrarians, the friendliest, most relevant and massively irreverent Big Data community on the whole of the entire world wide web. So, if you think you are up for it then we can be found on LinkedIn at this address:

https://www.linkedin.com/groups/8338976