yourfanat wrote: I am using another tool for Oracle developers - dbForge Studio for Oracle. This IDE has lots of usefull features, among them: oracle designer, code competion and formatter, query builder, debugger, profiler, erxport/import, reports and many others. The latest version supports Oracle 12C. More information here.
Cloud Computing
Conference & Expo
November 2-4, 2009 NYC
Register Today and SAVE !..

2008 West
Data Direct
SOA, WOA and Cloud Computing: The New Frontier for Data Services
Red Hat
The Opening of Virtualization
User Environment Management – The Third Layer of the Desktop
Cloud Computing for Business Agility
CMIS: A Multi-Vendor Proposal for a Service-Based Content Management Interoperability Standard
Freedom OSS
Practical SOA” Max Yankelevich
Architecting an Enterprise Service Router (ESR) – A Cost-Effective Way to Scale SOA Across the Enterprise
Return on Assests: Bringing Visibility to your SOA Strategy
Managing Hybrid Endpoint Environments
Game-Changing Technology for Enterprise Clouds and Applications
Click For 2008 West
Event Webcasts

2008 West
Get ‘Rich’ Quick: Rapid Prototyping for RIA with ZERO Server Code
Keynote Systems
Designing for and Managing Performance in the New Frontier of Rich Internet Applications
How Can AJAX Improve Homeland Security?
Beyond Widgets: What a RIA Platform Should Offer
REAs: Rich Enterprise Applications
Click For 2008 Event Webcasts
In many cases, the end of the year gives you time to step back and take stock of the last 12 months. This is when many of us take a hard look at what worked and what did not, complete performance reviews, and formulate plans for the coming year. For me, it is all of those things plus a time when I u...
Natural Language Processing | @BigDataExpo #BigData #Analytics #DataScience
Bridging the Gap Between Big Data and Big Information

Apophenia is the propensity to see patterns in random data.”  We encounter it all the time in the real world. Examples include gamblers who see patterns in how the cards are being dealt or investors who imagine patterns in the movement of certain stocks, or basketball fans who believe that their favorite player has the “hot hand.” But apophenia has no place in the world of data science, especially when data science is trying to help us make better decisions about critical things such as the quality of healthcare, where to allocate police resources, ensuring that our airplanes operate effectively or making investment decisions that determine our retirement readiness.

Understanding the differences between epiphany (a sudden, intuitive perception of or insight into the reality) and apophenia (the perception of or belief in connectedness among unrelated phenomena) is critical as data scientists build analytic models to quantify cause and effect. Regression Modeling is a great tool in helping to quantify cause and effect, but one still needs to leverage industry insights and sometimes just plain common sense to make sure that we are not trying to quantify “spurious relationships.” In the world of data science, one cannot automate out the importance of common sense. I think Craig Wilkey nails this distinction in this blog he wrote:

Apophenia is the propensity to see patterns in random data. It was first coined in 1958 by Klaus Conrad – a German neurologist and psychiatrist who, perhaps a little ironically, was attempting to identify early indicators of psychosis.

An apophany (an instance of apophenia) can best be defined in contrast to an epiphany. An epiphany is a moment of sudden and striking realization that leads a person to a greater degree of clarity in the nature of reality – a discovery of a truism, often hidden in plain sight. An apophany is having the experience of an epiphany, but you’re just plain wrong.

We’ve all heard some version of the old adage that correlation does not imply causation. It can be clearly demonstrated that in neighborhoods where there is an increase in ice cream consumption, there is a roughly equivalent spike in aggravated assault incidents. We’d be foolish to assume that eating ice cream makes people irrationally violent, but that doesn’t mean there’s nothing valuable to learn from this. When we broaden the lens a bit and include other variables, the connections become clearer.

In overpopulated urban environments, where there is a greater concentration of disenfranchised people – people who are statistically more likely to commit poverty crimes, and statistically less likely to have air-conditioned homes – heat waves usher in higher levels of frustration, lower levels of tolerance, and more people eating ice cream. We can also sharpen the focus by throwing in aggravation over public transit failures, brown-outs and black-outs, lower productivity, and countless other factors.

So, while enjoying tasty dairy products does not necessarily incite violence, the correlation between ice cream consumption and violence is not meaningless. Ice cream consumption analysis may indeed provide value as a leading indicator, or bellwether, of the potential for violent acts trending upward in a given community. If not a bellwether, it certainly is a valid correlation – as opposed to a simple coincidence.

The purpose of regression analysis is to identify those variables (referred to as independent variables) that help reveal valid correlations in the phenomena one is attempting to predict (the dependent variable).

Regression analysis is a tricky beast to harness. When the whole point is to find hidden correlations that may even defy intuitive understanding, it can be tempting to throw in the entire kitchen sink and see what comes out. The greatest perceived risk in that arises from patterns that may align, but are nevertheless invalid. These coincidences are referred to as ‘spurious relationships’.

If the patterns of some spurious relationship(s) happen to align with the patterns of other independent variables in a regression analysis model, the accuracy of the model will be impacted, and could be dramatically impacted.

It would be foolish to place any faith in all those quirky coincidences we always hear about with sports teams, for example. There is no reasonably conceivable way the first initial of the middle name of the first child born in some small town after the start of a sport’s season could predict the outcome of a team’s playoff standings – but I’d be genuinely surprised if there weren’t some spurious relationship to be found there.

On the other hand, we do have a valid argument for getting rid of the dramatic orchestra strike that foreshadows violent crime in movies, and replacing it with the sound of an ice cream truck.

How do we strike the balance between the desire to uncover hidden variables that provide valuable insight into trends, and the fear of creating an apophenic, potentially psychotic, regression analysis model?

My nearly two and a half decades of experience in IT have led me to the conclusion that the field suffers from rampant apopheniphobia: The irrational fear of finding ostensibly meaningful patterns in random data. (Yes, I did just make that word up. © Craig Wilkey, 2016)

Almost invariably, we simply do not push far enough. Should stock market analysis include things like weather patterns, celebrity news stories and grade school holidays? Absolutely! Classical stock market analysis techniques don’t work as well as they used to. Why? Frankly, we have a greater number of ignorant people playing the market. The proliferation of ‘Day Traders’ has crippled the old market truisms, because so many people who are affecting the market dynamics don’t have any classical training. The things that affect the moods and daily lives of ‘normal people’ need to be considered, because ‘normal people’ are far more active in the markets than they used to be. If they don’t play by the rules, then some of those rules simply cease to apply.

Apopheniphobia is fueled by fears of falling prey to spurious relationships. Who wants to be known as the person who unleashed a dangerously psychotic algorithm into the world? People think about the many statistical oddities they’ve come across, and it stunts their creative growth. For example, did you know there is a direct correlation between the per capita consumption of margarine and the divorce rate in Maine? Cheese consumption is far more dangerous than margarine consumption – it correlates with the number of people who die by becoming tangled in their bed sheets. (And you thought lactose intolerance was bad?) The number of people who drowned by falling into a pool also correlates with the number of films Nicholas Cage appeared in from 1999 through 2009. (Source)

In IT, we have a tendency to drive toward ‘proving’ clear, unambiguous relationships that quantify efforts, justify means and, more often than not, clearly align to our own preconceived notions. We want to be able to show clear lines of progression and indisputably direct relationships – we tend to believe anything less will not be trusted by those who hold the purse strings.

Our hyper-rational modes of thinking have a tendency to overshadow our creative imaginations – which, almost inevitably, leads to hampered understanding. Perhaps the greatest value of regression analysis is that it allows us to challenge our preconceived notions and learn something new. The greatest challenge with it is rarely throwing too much data at our models – it’s not having enough.

Yes, I know… We’re IT. We’re awash with data. We’re swimming in lakes of data and constantly inhaling the fumes of endless data exhaust. What we’re missing is the meaningful data extracted from unstructured information sources – in other words, the extraordinarily valuable information that’s locked away in language that has historically been inaccessible to machines – human language.

Estimates have been telling us for a decade or more that 80% of all information in a given organization is in the form of unstructured, human-readable text. I think there is nowhere that rings more true or significant than in trying to understand customer experience. I’d also argue that the majority of the most important service information is within that 80%.

Customer Experience Personalization absolutely depends on translating that human-readable text to machine-actionable data. When it comes to understanding and deriving value from actionable insights within our customer interactions, we must extract as much understanding from that unstructured text as possible and add it all to the other data in our regression models. Apopheniphobia be damned!

While it’s, admittedly, an oversimplification, it’s convenient to talk about two general approaches to extracting data from text. Text Analytics/Mining breaks the textual input into digestible chunks of string variables and uses statistical modeling techniques to find patterns in those variables. The ideal of Natural Language Processing is to develop a translation engine between human language and machine language. It uses some of the same statistical modeling approaches as Text Analytics, but goes much further by applying semantic and syntactic analysis to extract meaning, intention, sentiment and key concepts (among other things) included in the text.

Our best opportunity to achieve our vision of industry-leading Customer Experience Personalization is to take advantage of Natural Language Processing. This barely scratches the surface of what’s possible. Natural Language Processing will enable us to step aggressively toward extracting real meaning from the vast amount of otherwise machine-invisible, extraordinarily valuable content we have. Using that extracted meaning, in conjunction with our structured data points, will allow us to build truly valuable regression analysis models to understand our customers like never before. Keep pushing until the model breaks, then dial it back a scosche. That is the path to progress.

Apopheniphobia is the enemy of personalization and Customer Relationship Management. This is why I’ve decided to launch the Apopheniphobia Awareness Campaign. Please spread the word! I need to come up with a design for the lapel pin… Maybe a ribbon with as many digits of pi I can squeeze on it – with all the prime digits bolded? Maybe we should schedule a charity walk… We can follow the streets in alphabetical order.

The post Natural Language Processing: Bridging the Gap Between Big Data and Big Information appeared first on InFocus Blog | Dell EMC Services.

Read the original blog entry...

About William Schmarzo
Bill Schmarzo, author of “Big Data: Understanding How Data Powers Big Business” and “Big Data MBA: Driving Business Strategies with Data Science”, is responsible for setting strategy and defining the Big Data service offerings for Dell EMC’s Big Data Practice.

As a CTO within Dell EMC’s 2,000+ person consulting organization, he works with organizations to identify where and how to start their big data journeys. He’s written white papers, is an avid blogger and is a frequent speaker on the use of Big Data and data science to power an organization’s key business initiatives. He is a University of San Francisco School of Management (SOM) Executive Fellow where he teaches the “Big Data MBA” course. Bill also just completed a research paper on “Determining The Economic Value of Data”. Onalytica recently ranked Bill as #4 Big Data Influencer worldwide.

Bill has over three decades of experience in data warehousing, BI and analytics. Bill authored the Vision Workshop methodology that links an organization’s strategic business initiatives with their supporting data and analytic requirements. Bill serves on the City of San Jose’s Technology Innovation Board, and on the faculties of The Data Warehouse Institute and Strata.

Previously, Bill was vice president of Analytics at Yahoo where he was responsible for the development of Yahoo’s Advertiser and Website analytics products, including the delivery of “actionable insights” through a holistic user experience. Before that, Bill oversaw the Analytic Applications business unit at Business Objects, including the development, marketing and sales of their industry-defining analytic applications.

Bill holds a Masters Business Administration from University of Iowa and a Bachelor of Science degree in Mathematics, Computer Science and Business Administration from Coe College.

SOA World Latest Stories
The dynamic nature of the cloud means that change is a constant when it comes to modern cloud-based infrastructure. Delivering modern applications to end users, therefore, is a constantly shifting challenge. Delivery automation helps IT Ops teams ensure that apps are providing an optim...
"We started a Master of Science in business analytics - that's the hot topic. We serve the business community around San Francisco so we educate the working professionals and this is where they all want to be," explained Judy Lee, Associate Professor and Department Chair at Golden Gate...
There is a huge demand for responsive, real-time mobile and web experiences, but current architectural patterns do not easily accommodate applications that respond to events in real time. Common solutions using message queues or HTTP long-polling quickly lead to resiliency, scalability...
We call it DevOps but much of the time there’s a lot more discussion about the needs and concerns of developers than there is about other groups. There’s a focus on improved and less isolated developer workflows. There are many discussions around collaboration, continuous integration a...
Modern software design has fundamentally changed how we manage applications, causing many to turn to containers as the new virtual machine for resource management. As container adoption grows beyond stateless applications to stateful workloads, the need for persistent storage is founda...
"CA has been doing a lot of things in the area of DevOps. Now we have a complete set of tool sets in order to enable customers to go all the way from planning to development to testing down to release into the operations," explained Aruna Ravichandran, Vice President of Global Marketin...
Subscribe to the World's Most Powerful Newsletters
Subscribe to Our Rss Feeds & Get Your SYS-CON News Live!
Click to Add our RSS Feeds to the Service of Your Choice:
Google Reader or Homepage Add to My Yahoo! Subscribe with Bloglines Subscribe in NewsGator Online
myFeedster Add to My AOL Subscribe in Rojo Add 'Hugg' to Newsburst from CNET Kinja Digest View Additional SYS-CON Feeds
Publish Your Article! Please send it to editorial(at)!

Advertise on this site! Contact advertising(at)! 201 802-3021

SYS-CON Featured Whitepapers