Facebook GRAPH

I know everyone(at least pretend) uses Facebook and you should be noting that for last few weeks your Facebook homepage must me appearing different. Don’t say me no. It’s all due to new application from Facebook called as GRAPH. The is the 3rd revolutionary application from their stable. The other two were Newsfeed and Timeline.

So what’s Graph, it’s a search application (remember the various graph algorithms we studied in our days of Engineering) and it works in the same fashion. Suppose you wanted to search anything about let’s say about Tiger. Traditional search solutions like Google will return the same results for everyone unless you signed in google and they track your search using the cookies. Facebook GRAPH will use your friends list (Deloitte calls it as Networking) to get everything related to Tiger. If someone in your network had visited Bandipur and commented that its better than Mudumalai, it’s going to influence your decision. Some of us stalk behind people to get info, no need going forward.

You can also have a little bit more insight about graph based search from the below link,

Regarding the privacy, yes Facebook allows to have a control on the visibility/availability of your profile for search.

Marriage between Big data and Cloud

When I was starting my career in IT 3 years back, everyone was talking about cloud and it was called as the next big thing in IT. But still it hadn’t made the anticipated impact. Now the industry is buzzing ‘Big Data’ and I hope it don’t be just a hype.

Talking ‘Big Data’ means the application solutions developed on Hadoop need not be restricted only to handle huge volumes of Petabytes of data. The best of big data can also be used for databases of smaller size as every organization is not going to handle huge volume of data. So if we think who will be using the Big Data solutions for small volume of data, the focus will be definitely on those who are actively using cloud for their business.

Cloud came into picture for variety of reasons like green IT, affordability and its robust nature. The primary target audience of cloud were mostly the SMB (Small and Medium scale Business) units who were previously finding difficult to use IT solutions due to cost constraints. All early starters in cloud have tremendously benefited and have seen the benefits in due course. Moving away from SMB’s, we should also remember the fact that cloud also has capabilities to store petabytes of data. Eg: Amazon’s data centers providing EC2 cloud service is a perfect example for this.

To provide Big Data solutions like analytic services to SMB’s a marriage of convenience between cloud and big data has to be considered to provide best in value to all companies. and with its marriage, Big data will extract high quality information needed for business improvement. With Hadoop being open source the companies need to work out their finances only for cloud. Amazon provides hadoop in cloud but it is not providing any other services to derive value from the data.

The IT services companies are currently providing most of the industry specific solutions as a template in their respective cloud environments. To point a few REPUBLIC from Hexaware, Ion from TCS are some of them. It will be better if we include big data in their offerings.

All readers, kindly let me know your thoughts also on this.

Data Pollution

We know different kinds of pollution but there is a new kind of pollution that doesn’t affect the environment but seriously hampers our ability to take decision. We can name it as data pollution. With cheap data storage devices we are lethargic in maintaining quality of data. Even though we strive to maintain quality of data in the warehouses through various control measures, still the volume of data is huge causing considerable amount of pollution.

Data pollution is caused by poorly formed data which is of less importance to humans. Recently we faced a similar problem where one user entered alias name for our currency as Indian rupee and another as Indian national rupee. The above mentioned is a perfect example of data pollution.

We BI people always take pride in fancy reports which we developed for people in the higher management. Suppose the same report displays the amount for a report based on the alias name of currency instead of currency codes definitely there will be amount mismatch which destroys the credibility of the entire report. The business impact varies based on the variation or difference in the amount values.

I think growth in volume of data is the only serious competitor to Moore’s law and reliability of data is very high these days and even the Indian government has to change its IIP (Index of Industrial Production) figures as there was data pollution causing huge embarrassment. So kindly takes steps to control pollution of data.

Kindly share your thoughts on how data pollution has corrupted your data.

Pushing data to Hyperion Planning Applications from Informatica 9.1

With EOL notice from Informatica for its 8.x versions, to get continuous support from Informatica it’s mandatory to migrate to 9.x versions.

Hyperion Planning application

On analyzing our old environment in 8.1, we found that 9.1 doesn’t support Hyperion Planning Application connection but we need to push data to planning application. We need to replace the existing planning adapter targets in the mapping with a delimited flat file. This flat file will act as source for the data push to Hyperion Planning.

We can push data to planning application by two methods,

1. Java code which makes use of Hyperion planning library files.
2. Through ODI (Oracle Data Integrator)

Method: 1 (Using Java Code)

It involves two steps,

1. Java code

We first need to develop a java file which makes use of HSP.jar library file of Hyperion planning application. The program flow includes locking a dimension before doing a data push, checking for UDA and finally it saves the dimension member using saveMember() function.

1. Batch file

This batch file should call the above java file and the parameters are planning application name, server, RMI port no, user name, password and delimiter.

In post success command of a session, we will call this batch file along with the parameters. The problem was that it took more time to complete and it was also noted that throughput was very less while saving the records in planning application using saveMember().

Method: 2 (Using ODI)

The reason why we went on to implement ODI is that we don’t need to get new licenses if we are using Hyperion suite of applications.

Steps in ODI Designer:

1. Install and configure ODI, in the same server where Informatica is installed.
2. Reverse engineer the Hyperion planning knowledge module.
3. Create planning models (targets) and file models.
4. Create interfaces for all the UDA’s, attributes and dimensions and specify log file names.
5. Create packages for each dimension say Customer, the flow should have first its attribute interfaces, UDA’s if applicable and then customer interface.
6. Create scenario’s (Scenarios will be called from batch file) and publish the package.

Steps in Topology Manager:

We need to create physical schemas and context for the different planning environments. Generally we would be having two planning applications viz budgeting and forecasting for planning DEV, UACC and PROD.

Scenario’s can be started from command prompt and the syntax is,

startscen <<scenario name>> <<version no>> <<context name>> <<trace level>>

A batch file needs to written to call the scenarios from command task

set EXECPATH=D:\Oracle\ODI\oracledi\bin

cd /d “%EXECPATH%”

call startscen.bat %SCENARIONAME% %VERSION% %CONTEXT% %LOGLEVEL% %TRACELEVEL%

In Informatica workflow designer, worklet variable to specify the context name at run time needs to be created and we will be passing the values in parameter file. Call the batch file created specifically for this purpose in post session command task.

Conclusion

On comparing these both methods, through ODI we found that time taken for data load was consistent and was near to the time taken by Hyperion planning adapter in Informatica 8.1 than through JAVA code.

Diesel cars are not cheap..why?? Look below

With the price gap between unregulated petrol and subsidized diesel, we see many people are buying diesel powered cars and their perception is that diesel is cheap and so they will be spending less. Due to this companies are aggressively promoting so called highly pollutant diesel like ford figo, suzuki swift,etc,. But if you look into lot facts its not.

1. Diesel engine cars are more expensive than traditional petrol driven ones. Normally the gap is around 2 lakhs.

2. If we buy cars on loans, we have to pay more interest for them as they are expensive.

3. Traditionally diesel powered cars do have a high maintenance cost.

4. If you drive a diesel powered car for around 20k kms per year, you would be saving around 30,000 rupees with current prices and with assumed mileage of 20kms/litre.

5. You should drive this car for another 5 more years to reap the benefits of buying a diesel car which means by this time car reaches its shell life. Now you will boggled down by the maintenance expenses which will spiral upwards.

6. Last one is important Government of India proposed to levy a excise duty of 60000 rupees for diesel cars which will be huge for small car buyers where as the diesel guzzler SUV buyers don’t worry about this.

A futuristic approach-Moving Gaming to Cloud

I was in constant fear due to my aging laptop as with the new games it needed a upgrade to support them. Everyone knows that upgrading a laptop is an expensive affair. I was wondering why we can’t find a solution to this with available technologies. I think the answer is cloud computing. Nowadays we see very large implementations being moved into the cloud space and infrastructure is being used in a wise manner. We also need to know that the popular games in facebook like Farmville can also be categorized as games in cloud.

If we move all our games to cloud then it’s just a plug and play type. All we need is switch on your laptop connect to net, login and start the action. The companies like EA sports, Microsoft also stand to benefit as if the gaming is moved to cloud it will totally eradicate piracy. Even from mobiles which have less computing power compared to desktops can be used as a medium to play power packed games.

But I guess it will take time to move gaming into cloud as still the penetration of broadband internet is low and initially it may face a stiff resistance from the gamers as even the fortune 500 companies are yet to move their entire IT infrastructure to cloud. But it will be a paradigm shift as the industry will coin a new term called GaaS which Is an acronym for Gaming as a Service.

Importance of Social Media

For almost all PR agencies today to showcassing their clients in spotlight in social media like Twitter,LinkedIn and Facebook has become a necessity one.

Few years back before social media entered actually entered, people would socialize in marriages, corporate events and summits and build their connections. But now its not the case with social media entering into the living rooms. Comedies centering the impact of social media like father asking his son to come for dinner through facebook has become a real one.

Once a product is lauched in the market, immediately the social media is flooded with 360 degree analysis of the product and if negatives aer more the product will doom to extinct soon. No doubt in that. If companies doesn’t manage their brands in social media it is a disaster for them and the same happened for Kingfisher during its recent fleet restructuring.

Need of Hadoop Distributed File System

People would always think how the organizations like Yahoo, Google, Facebook store large amounts of data of the users. We should take a note that Facebook stores more photos than Google’s Picassa. Any guesses??

The answer is Hadoop and it is a way to store large amounts of data in petabytes and zettabytes. This storage system is called as Hadoop Distributed File System. Hadoop was developed by Doug Cutting based on ideas suggested by Google’s papers. Mostly we get large amounts of machine generated data. For example, the Large Hadron Collider to study the origins of universe produces 15 petabytes of data every year for each experiment carried out.

The next thing which comes to our mind is how quick we can access these large amounts of data. Hadoop also uses Map Reduce. It follows ‘Divide and Conquer’. The data is organized as key value pairs. It processes the entire data that is spread across countless number of systems in parallel chunks from a single node. Then it will sort and process the collected data.

With a standard PC server, Hadoop will connect to all the servers and distributes the data files across these nodes. It used all these nodes as one large file system to store and process the data , making it a 100% unadulterated distributed file system. Extra nodes can be added if data reaches the maximum installed capacity making the setup highly scalable. It is very cheap as it is open source and doesn’t require special processors like used in traditional servers. Hadoop is also one of the NoSQL implementations.

The Tennessee Valley Authority uses smart-grid field devices to collect data on its power-transmission lines and facilities across the country. These sensors send in data at a rate of 30 times per second – at that rate, the TVA estimates it will have half a petabyte of data archived within a few years. TVA  uses Hadoop to store and analyse data. Our own Power Grid Corporation of India intends to install these smart devices in their grids for collecting data to reduce transmission losses. It is better they also emulate TVA.

Interbank Mobile Payment Service

The newly launched IBPS (Interbank Mobile Payment Service) is a innovative mechanism brought up by National Payments Corporation has a lot of potential in providing banking services to the people who are living at the bottom of the pyramid.I believe this is the only initiative taken up to reach bottom of pyramid where fortune is present as stated by the late Management Guru C.K.Prahlad. With the amount of mobile subscribers in our country it provides yet another platform for banking among the masses.

All you have to do to use this innovative service is get this application installed in your mobile phone which primarily works through SMS or USSD protocal. MMID developed by IIT Chennai combines your mobile number, bank account number and a routing code for transactions .Get yourself registered with your bank and get your unique MMID. The person at the other end also should get the same. To send money just enter the person’s MMID and mobile number, the transaction is complete. The cost is also very low.Believe it or not only normal SMS charges are applicable.

I hope after this Banking for Masses will become real and there is scope for transferring amounts to workers for central government’s flagship scheme NREGES for rural people.As of now 18 banks are providing this service.

Partition Over in Oracle

Quick data retrieval is what we expect, Oracle in 10g has provided some new keywords that can be used to reduce the time when group by is used.Below are the queries which are self explanatory.

The query is used to find employees whose salaries are greater than average salary of the department.

Normal Query:

select e.empid,e.salary,e1.avgsal from emp e join (select deptid,avg(salary) avgsal from emp group by deptid)e1 on e.deptid=e1.deptid where e.salary>e1.avgsal

New Query:

select empid,salary,deptid,avg(salary) over (partition by deptid) avgsal from emp e