<?xml version="1.0" encoding="UTF-8"?>
<rss version="2.0"
	xmlns:content="http://purl.org/rss/1.0/modules/content/"
	xmlns:wfw="http://wellformedweb.org/CommentAPI/"
	xmlns:dc="http://purl.org/dc/elements/1.1/"
	xmlns:atom="http://www.w3.org/2005/Atom"
	xmlns:sy="http://purl.org/rss/1.0/modules/syndication/"
	xmlns:slash="http://purl.org/rss/1.0/modules/slash/"
	>

<channel>
	<title>Software Secret Weapons</title>
	<atom:link href="http://www.softwaresecretweapons.com/jspwiki/feed" rel="self" type="application/rss+xml" />
	<link>http://www.softwaresecretweapons.com/jspwiki</link>
	<description></description>
	<lastBuildDate>Mon, 24 Oct 2011 18:25:52 +0000</lastBuildDate>
	<generator>http://wordpress.org/?v=2.9.2</generator>
	<language>en</language>
	<sy:updatePeriod>hourly</sy:updatePeriod>
	<sy:updateFrequency>1</sy:updateFrequency>
			<item>
		<title>Stevey&#8217;s Google Platforms Rant</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/steveys-google-platforms-rant</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/steveys-google-platforms-rant#comments</comments>
		<pubDate>Mon, 24 Oct 2011 18:22:46 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=589</guid>
		<description><![CDATA[this article is cached locally for the public records as the original will likely be removed eventually; Steve is know for his rants; he posted Googles Secret Weapon rant back in 2005; it is now removed from the source, but I kept a copy&#8230;

I was at Amazon for about six and a half years, and [...]]]></description>
			<content:encoded><![CDATA[<p>this article is cached locally for the public records as the <a href='https://plus.google.com/112678702228711889851/posts/eVeouesvaVX'>original</a> will likely be removed eventually; Steve is know for his rants; he posted <a href='http://www.softwaresecretweapons.com/jspwiki/googlessecretweapon'>Googles Secret Weapon</a> rant back in 2005; it is now removed from the source, but I kept a copy&#8230;</p>
<hr />
<p>I was at Amazon for about six and a half years, and now I&#8217;ve been at Google for that long. One thing that struck me immediately about the two companies &#8212; an impression that has been reinforced almost daily &#8212; is that Amazon does everything wrong, and Google does everything right. Sure, it&#8217;s a sweeping generalization, but a surprisingly accurate one. It&#8217;s pretty crazy. There are probably a hundred or even two hundred different ways you can compare the two companies, and Google is superior in all but three of them, if I recall correctly. I actually did a spreadsheet at one point but Legal wouldn&#8217;t let me show it to anyone, even though recruiting loved it.</p>
<p>I mean, just to give you a very brief taste: Amazon&#8217;s recruiting process is fundamentally flawed by having teams hire for themselves, so their hiring bar is incredibly inconsistent across teams, despite various efforts they&#8217;ve made to level it out. And their operations are a mess; they don&#8217;t really have SREs and they make engineers pretty much do everything, which leaves almost no time for coding &#8211; though again this varies by group, so it&#8217;s luck of the draw. They don&#8217;t give a single shit about charity or helping the needy or community contributions or anything like that. Never comes up there, except maybe to laugh about it. Their facilities are dirt-smeared cube farms without a dime spent on decor or common meeting areas. Their pay and benefits suck, although much less so lately due to local competition from Google and Facebook. But they don&#8217;t have any of our perks or extras &#8212; they just try to match the offer-letter numbers, and that&#8217;s the end of it. Their code base is a disaster, with no engineering standards whatsoever except what individual teams choose to put in place.</p>
<p>To be fair, they do have a nice versioned-library system that we really ought to emulate, and a nice publish-subscribe system that we also have no equivalent for. But for the most part they just have a bunch of crappy tools that read and write state machine information into relational databases. We wouldn&#8217;t take most of it even if it were free.</p>
<p>I think the pubsub system and their library-shelf system were two out of the grand total of three things Amazon does better than google.</p>
<p>I guess you could make an argument that their bias for launching early and iterating like mad is also something they do well, but you can argue it either way. They prioritize launching early over everything else, including retention and engineering discipline and a bunch of other stuff that turns out to matter in the long run. So even though it&#8217;s given them some competitive advantages in the marketplace, it&#8217;s created enough other problems to make it something less than a slam-dunk.</p>
<p>But there&#8217;s one thing they do really really well that pretty much makes up for ALL of their political, philosophical and technical screw-ups.</p>
<p>Jeff Bezos is an infamous micro-manager. He micro-manages every single pixel of Amazon&#8217;s retail site. He hired Larry Tesler, Apple&#8217;s Chief Scientist and probably the very most famous and respected human-computer interaction expert in the entire world, and then ignored every goddamn thing Larry said for three years until Larry finally &#8212; wisely &#8212; left the company. Larry would do these big usability studies and demonstrate beyond any shred of doubt that nobody can understand that frigging website, but Bezos just couldn&#8217;t let go of those pixels, all those millions of semantics-packed pixels on the landing page. They were like millions of his own precious children. So they&#8217;re all still there, and Larry is not.</p>
<p>Micro-managing isn&#8217;t that third thing that Amazon does better than us, by the way. I mean, yeah, they micro-manage really well, but I wouldn&#8217;t list it as a strength or anything. I&#8217;m just trying to set the context here, to help you understand what happened. We&#8217;re talking about a guy who in all seriousness has said on many public occasions that people should be paying him to work at Amazon. He hands out little yellow stickies with his name on them, reminding people &#8220;who runs the company&#8221; when they disagree with him. The guy is a regular&#8230; well, Steve Jobs, I guess. Except without the fashion or design sense. Bezos is super smart; don&#8217;t get me wrong. He just makes ordinary control freaks look like stoned hippies.</p>
<p>So one day Jeff Bezos issued a mandate. He&#8217;s doing that all the time, of course, and people scramble like ants being pounded with a rubber mallet whenever it happens. But on one occasion &#8212; back around 2002 I think, plus or minus a year &#8212; he issued a mandate that was so out there, so huge and eye-bulgingly ponderous, that it made all of his other mandates look like unsolicited peer bonuses.</p>
<p>His Big Mandate went something along these lines:</p>
<p>1) All teams will henceforth expose their data and functionality through service interfaces.</p>
<p>2) Teams must communicate with each other through these interfaces.</p>
<p>3) There will be no other form of interprocess communication allowed: no direct linking, no direct reads of another team&#8217;s data store, no shared-memory model, no back-doors whatsoever. The only communication allowed is via service interface calls over the network.</p>
<p>4) It doesn&#8217;t matter what technology they use. HTTP, Corba, Pubsub, custom protocols &#8212; doesn&#8217;t matter. Bezos doesn&#8217;t care.</p>
<p>5) All service interfaces, without exception, must be designed from the ground up to be externalizable. That is to say, the team must plan and design to be able to expose the interface to developers in the outside world. No exceptions.</p>
<p>6) Anyone who doesn&#8217;t do this will be fired.</p>
<p>7) Thank you; have a nice day!</p>
<p>Ha, ha! You 150-odd ex-Amazon folks here will of course realize immediately that #7 was a little joke I threw in, because Bezos most definitely does not give a shit about your day.</p>
<p>#6, however, was quite real, so people went to work. Bezos assigned a couple of Chief Bulldogs to oversee the effort and ensure forward progress, headed up by Uber-Chief Bear Bulldog Rick Dalzell. Rick is an ex-Armgy Ranger, West Point Academy graduate, ex-boxer, ex-Chief Torturer slash CIO at Wal*Mart, and is a big genial scary man who used the word &#8220;hardened interface&#8221; a lot. Rick was a walking, talking hardened interface himself, so needless to say, everyone made LOTS of forward progress and made sure Rick knew about it.</p>
<p>Over the next couple of years, Amazon transformed internally into a service-oriented architecture. They learned a tremendous amount while effecting this transformation. There was lots of existing documentation and lore about SOAs, but at Amazon&#8217;s vast scale it was about as useful as telling Indiana Jones to look both ways before crossing the street. Amazon&#8217;s dev staff made a lot of discoveries along the way. A teeny tiny sampling of these discoveries included:</p>
<p>- pager escalation gets way harder, because a ticket might bounce through 20 service calls before the real owner is identified. If each bounce goes through a team with a 15-minute response time, it can be hours before the right team finally finds out, unless you build a lot of scaffolding and metrics and reporting.</p>
<p>- every single one of your peer teams suddenly becomes a potential DOS attacker. Nobody can make any real forward progress until very serious quotas and throttling are put in place in every single service.</p>
<p>- monitoring and QA are the same thing. You&#8217;d never think so until you try doing a big SOA. But when your service says &#8220;oh yes, I&#8217;m fine&#8221;, it may well be the case that the only thing still functioning in the server is the little component that knows how to say &#8220;I&#8217;m fine, roger roger, over and out&#8221; in a cheery droid voice. In order to tell whether the service is actually responding, you have to make individual calls. The problem continues recursively until your monitoring is doing comprehensive semantics checking of your entire range of services and data, at which point it&#8217;s indistinguishable from automated QA. So they&#8217;re a continuum.</p>
<p>- if you have hundreds of services, and your code MUST communicate with other groups&#8217; code via these services, then you won&#8217;t be able to find any of them without a service-discovery mechanism. And you can&#8217;t have that without a service registration mechanism, which itself is another service. So Amazon has a universal service registry where you can find out reflectively (programmatically) about every service, what its APIs are, and also whether it is currently up, and where.</p>
<p>- debugging problems with someone else&#8217;s code gets a LOT harder, and is basically impossible unless there is a universal standard way to run every service in a debuggable sandbox.</p>
<p>That&#8217;s just a very small sample. There are dozens, maybe hundreds of individual learnings like these that Amazon had to discover organically. There were a lot of wacky ones around externalizing services, but not as many as you might think. Organizing into services taught teams not to trust each other in most of the same ways they&#8217;re not supposed to trust external developers.</p>
<p>This effort was still underway when I left to join Google in mid-2005, but it was pretty far advanced. From the time Bezos issued his edict through the time I left, Amazon had transformed culturally into a company that thinks about everything in a services-first fashion. It is now fundamental to how they approach all designs, including internal designs for stuff that might never see the light of day externally.</p>
<p>At this point they don&#8217;t even do it out of fear of being fired. I mean, they&#8217;re still afraid of that; it&#8217;s pretty much part of daily life there, working for the Dread Pirate Bezos and all. But they do services because they&#8217;ve come to understand that it&#8217;s the Right Thing. There are without question pros and cons to the SOA approach, and some of the cons are pretty long. But overall it&#8217;s the right thing because SOA-driven design enables Platforms.</p>
<p>That&#8217;s what Bezos was up to with his edict, of course. He didn&#8217;t (and doesn&#8217;t) care even a tiny bit about the well-being of the teams, nor about what technologies they use, nor in fact any detail whatsoever about how they go about their business unless they happen to be screwing up. But Bezos realized long before the vast majority of Amazonians that Amazon needs to be a platform.</p>
<p>You wouldn&#8217;t really think that an online bookstore needs to be an extensible, programmable platform. Would you?</p>
<p>Well, the first big thing Bezos realized is that the infrastructure they&#8217;d built for selling and shipping books and sundry could be transformed an excellent repurposable computing platform. So now they have the Amazon Elastic Compute Cloud, and the Amazon Elastic MapReduce, and the Amazon Relational Database Service, and a whole passel&#8217; o&#8217; other services browsable at aws.amazon.com. These services host the backends for some pretty successful companies, reddit being my personal favorite of the bunch.</p>
<p>The other big realization he had was that he can&#8217;t always build the right thing. I think Larry Tesler might have struck some kind of chord in Bezos when he said his mom couldn&#8217;t use the goddamn website. It&#8217;s not even super clear whose mom he was talking about, and doesn&#8217;t really matter, because nobody&#8217;s mom can use the goddamn website. In fact I myself find the website disturbingly daunting, and I worked there for over half a decade. I&#8217;ve just learned to kinda defocus my eyes and concentrate on the million or so pixels near the center of the page above the fold.</p>
<p>I&#8217;m not really sure how Bezos came to this realization &#8212; the insight that he can&#8217;t build one product and have it be right for everyone. But it doesn&#8217;t matter, because he gets it. There&#8217;s actually a formal name for this phenomenon. It&#8217;s called Accessibility, and it&#8217;s the most important thing in the computing world.</p>
<p>The. Most. Important. Thing.</p>
<p>If you&#8217;re sorta thinking, &#8220;huh? You mean like, blind and deaf people Accessibility?&#8221; then you&#8217;re not alone, because I&#8217;ve come to understand that there are lots and LOTS of people just like you: people for whom this idea does not have the right Accessibility, so it hasn&#8217;t been able to get through to you yet. It&#8217;s not your fault for not understanding, any more than it would be your fault for being blind or deaf or motion-restricted or living with any other disability. When software &#8212; or idea-ware for that matter &#8212; fails to be accessible to anyone for any reason, it is the fault of the software or of the messaging of the idea. It is an Accessibility failure.</p>
<p>Like anything else big and important in life, Accessibility has an evil twin who, jilted by the unbalanced affection displayed by their parents in their youth, has grown into an equally powerful Arch-Nemesis (yes, there&#8217;s more than one nemesis to accessibility) named Security. And boy howdy are the two ever at odds.</p>
<p>But I&#8217;ll argue that Accessibility is actually more important than Security because dialing Accessibility to zero means you have no product at all, whereas dialing Security to zero can still get you a reasonably successful product such as the Playstation Network.</p>
<p>So yeah. In case you hadn&#8217;t noticed, I could actually write a book on this topic. A fat one, filled with amusing anecdotes about ants and rubber mallets at companies I&#8217;ve worked at. But I will never get this little rant published, and you&#8217;ll never get it read, unless I start to wrap up.</p>
<p>That one last thing that Google doesn&#8217;t do well is Platforms. We don&#8217;t understand platforms. We don&#8217;t &#8220;get&#8221; platforms. Some of you do, but you are the minority. This has become painfully clear to me over the past six years. I was kind of hoping that competitive pressure from Microsoft and Amazon and more recently Facebook would make us wake up collectively and start doing universal services. Not in some sort of ad-hoc, half-assed way, but in more or less the same way Amazon did it: all at once, for real, no cheating, and treating it as our top priority from now on.</p>
<p>But no. No, it&#8217;s like our tenth or eleventh priority. Or fifteenth, I don&#8217;t know. It&#8217;s pretty low. There are a few teams who treat the idea very seriously, but most teams either don&#8217;t think about it all, ever, or only a small percentage of them think about it in a very small way.</p>
<p>It&#8217;s a big stretch even to get most teams to offer a stubby service to get programmatic access to their data and computations. Most of them think they&#8217;re building products. And a stubby service is a pretty pathetic service. Go back and look at that partial list of learnings from Amazon, and tell me which ones Stubby gives you out of the box. As far as I&#8217;m concerned, it&#8217;s none of them. Stubby&#8217;s great, but it&#8217;s like parts when you need a car.</p>
<p>A product is useless without a platform, or more precisely and accurately, a platform-less product will always be replaced by an equivalent platform-ized product.</p>
<p>Google+ is a prime example of our complete failure to understand platforms from the very highest levels of executive leadership (hi Larry, Sergey, Eric, Vic, howdy howdy) down to the very lowest leaf workers (hey yo). We all don&#8217;t get it. The Golden Rule of platforms is that you Eat Your Own Dogfood. The Google+ platform is a pathetic afterthought. We had no API at all at launch, and last I checked, we had one measly API call. One of the team members marched in and told me about it when they launched, and I asked: &#8220;So is it the Stalker API?&#8221; She got all glum and said &#8220;Yeah.&#8221; I mean, I was joking, but no&#8230; the only API call we offer is to get someone&#8217;s stream. So I guess the joke was on me.</p>
<p>Microsoft has known about the Dogfood rule for at least twenty years. It&#8217;s been part of their culture for a whole generation now. You don&#8217;t eat People Food and give your developers Dog Food. Doing that is simply robbing your long-term platform value for short-term successes. Platforms are all about long-term thinking.</p>
<p>Google+ is a knee-jerk reaction, a study in short-term thinking, predicated on the incorrect notion that Facebook is successful because they built a great product. But that&#8217;s not why they are successful. Facebook is successful because they built an entire constellation of products by allowing other people to do the work. So Facebook is different for everyone. Some people spend all their time on Mafia Wars. Some spend all their time on Farmville. There are hundreds or maybe thousands of different high-quality time sinks available, so there&#8217;s something there for everyone.</p>
<p>Our Google+ team took a look at the aftermarket and said: &#8220;Gosh, it looks like we need some games. Let&#8217;s go contract someone to, um, write some games for us.&#8221; Do you begin to see how incredibly wrong that thinking is now? The problem is that we are trying to predict what people want and deliver it for them.</p>
<p>You can&#8217;t do that. Not really. Not reliably. There have been precious few people in the world, over the entire history of computing, who have been able to do it reliably. Steve Jobs was one of them. We don&#8217;t have a Steve Jobs here. I&#8217;m sorry, but we don&#8217;t.</p>
<p>Larry Tesler may have convinced Bezos that he was no Steve Jobs, but Bezos realized that he didn&#8217;t need to be a Steve Jobs in order to provide everyone with the right products: interfaces and workflows that they liked and felt at ease with. He just needed to enable third-party developers to do it, and it would happen automatically.</p>
<p>I apologize to those (many) of you for whom all this stuff I&#8217;m saying is incredibly obvious, because yeah. It&#8217;s incredibly frigging obvious. Except we&#8217;re not doing it. We don&#8217;t get Platforms, and we don&#8217;t get Accessibility. The two are basically the same thing, because platforms solve accessibility. A platform is accessibility.</p>
<p>So yeah, Microsoft gets it. And you know as well as I do how surprising that is, because they don&#8217;t &#8220;get&#8221; much of anything, really. But they understand platforms as a purely accidental outgrowth of having started life in the business of providing platforms. So they have thirty-plus years of learning in this space. And if you go to msdn.com, and spend some time browsing, and you&#8217;ve never seen it before, prepare to be amazed. Because it&#8217;s staggeringly huge. They have thousands, and thousands, and THOUSANDS of API calls. They have a HUGE platform. Too big in fact, because they can&#8217;t design for squat, but at least they&#8217;re doing it.</p>
<p>Amazon gets it. Amazon&#8217;s AWS (aws.amazon.com) is incredible. Just go look at it. Click around. It&#8217;s embarrassing. We don&#8217;t have any of that stuff.</p>
<p>Apple gets it, obviously. They&#8217;ve made some fundamentally non-open choices, particularly around their mobile platform. But they understand accessibility and they understand the power of third-party development and they eat their dogfood. And you know what? They make pretty good dogfood. Their APIs are a hell of a lot cleaner than Microsoft&#8217;s, and have been since time immemorial.</p>
<p>Facebook gets it. That&#8217;s what really worries me. That&#8217;s what got me off my lazy butt to write this thing. I hate blogging. I hate&#8230; plussing, or whatever it&#8217;s called when you do a massive rant in Google+ even though it&#8217;s a terrible venue for it but you do it anyway because in the end you really do want Google to be successful. And I do! I mean, Facebook wants me there, and it&#8217;d be pretty easy to just go. But Google is home, so I&#8217;m insisting that we have this little family intervention, uncomfortable as it might be.</p>
<p>After you&#8217;ve marveled at the platform offerings of Microsoft and Amazon, and Facebook I guess (I didn&#8217;t look because I didn&#8217;t want to get too depressed), head over to developers.google.com and browse a little. Pretty big difference, eh? It&#8217;s like what your fifth-grade nephew might mock up if he were doing an assignment to demonstrate what a big powerful platform company might be building if all they had, resource-wise, was one fifth grader.</p>
<p>Please don&#8217;t get me wrong here &#8212; I know for a fact that the dev-rel team has had to FIGHT to get even this much available externally. They&#8217;re kicking ass as far as I&#8217;m concerned, because they DO get platforms, and they are struggling heroically to try to create one in an environment that is at best platform-apathetic, and at worst often openly hostile to the idea.</p>
<p>I&#8217;m just frankly describing what developers.google.com looks like to an outsider. It looks childish. Where&#8217;s the Maps APIs in there for Christ&#8217;s sake? Some of the things in there are labs projects. And the APIs for everything I clicked were&#8230; they were paltry. They were obviously dog food. Not even good organic stuff. Compared to our internal APIs it&#8217;s all snouts and horse hooves.</p>
<p>And also don&#8217;t get me wrong about Google+. They&#8217;re far from the only offenders. This is a cultural thing. What we have going on internally is basically a war, with the underdog minority Platformers fighting a more or less losing battle against the Mighty Funded Confident Producters.</p>
<p>Any teams that have successfully internalized the notion that they should be externally programmable platforms from the ground up are underdogs &#8212; Maps and Docs come to mind, and I know GMail is making overtures in that direction. But it&#8217;s hard for them to get funding for it because it&#8217;s not part of our culture. Maestro&#8217;s funding is a feeble thing compared to the gargantuan Microsoft Office programming platform: it&#8217;s a fluffy rabbit versus a T-Rex. The Docs team knows they&#8217;ll never be competitive with Office until they can match its scripting facilities, but they&#8217;re not getting any resource love. I mean, I assume they&#8217;re not, given that Apps Script only works in Spreadsheet right now, and it doesn&#8217;t even have keyboard shortcuts as part of its API. That team looks pretty unloved to me.</p>
<p>Ironically enough, Wave was a great platform, may they rest in peace. But making something a platform is not going to make you an instant success. A platform needs a killer app. Facebook &#8212; that is, the stock service they offer with walls and friends and such &#8212; is the killer app for the Facebook Platform. And it is a very serious mistake to conclude that the Facebook App could have been anywhere near as successful without the Facebook Platform.</p>
<p>You know how people are always saying Google is arrogant? I&#8217;m a Googler, so I get as irritated as you do when people say that. We&#8217;re not arrogant, by and large. We&#8217;re, like, 99% Arrogance-Free. I did start this post &#8212; if you&#8217;ll reach back into distant memory &#8212; by describing Google as &#8220;doing everything right&#8221;. We do mean well, and for the most part when people say we&#8217;re arrogant it&#8217;s because we didn&#8217;t hire them, or they&#8217;re unhappy with our policies, or something along those lines. They&#8217;re inferring arrogance because it makes them feel better.</p>
<p>But when we take the stance that we know how to design the perfect product for everyone, and believe you me, I hear that a lot, then we&#8217;re being fools. You can attribute it to arrogance, or naivete, or whatever &#8212; it doesn&#8217;t matter in the end, because it&#8217;s foolishness. There IS no perfect product for everyone.</p>
<p>And so we wind up with a browser that doesn&#8217;t let you set the default font size. Talk about an affront to Accessibility. I mean, as I get older I&#8217;m actually going blind. For real. I&#8217;ve been nearsighted all my life, and once you hit 40 years old you stop being able to see things up close. So font selection becomes this life-or-death thing: it can lock you out of the product completely. But the Chrome team is flat-out arrogant here: they want to build a zero-configuration product, and they&#8217;re quite brazen about it, and Fuck You if you&#8217;re blind or deaf or whatever. Hit Ctrl-+ on every single page visit for the rest of your life.</p>
<p>It&#8217;s not just them. It&#8217;s everyone. The problem is that we&#8217;re a Product Company through and through. We built a successful product with broad appeal &#8212; our search, that is &#8212; and that wild success has biased us.</p>
<p>Amazon was a product company too, so it took an out-of-band force to make Bezos understand the need for a platform. That force was their evaporating margins; he was cornered and had to think of a way out. But all he had was a bunch of engineers and all these computers&#8230; if only they could be monetized somehow&#8230; you can see how he arrived at AWS, in hindsight.</p>
<p>Microsoft started out as a platform, so they&#8217;ve just had lots of practice at it.</p>
<p>Facebook, though: they worry me. I&#8217;m no expert, but I&#8217;m pretty sure they started off as a Product and they rode that success pretty far. So I&#8217;m not sure exactly how they made the transition to a platform. It was a relatively long time ago, since they had to be a platform before (now very old) things like Mafia Wars could come along.</p>
<p>Maybe they just looked at us and asked: &#8220;How can we beat Google? What are they missing?&#8221;</p>
<p>The problem we face is pretty huge, because it will take a dramatic cultural change in order for us to start catching up. We don&#8217;t do internal service-oriented platforms, and we just as equally don&#8217;t do external ones. This means that the &#8220;not getting it&#8221; is endemic across the company: the PMs don&#8217;t get it, the engineers don&#8217;t get it, the product teams don&#8217;t get it, nobody gets it. Even if individuals do, even if YOU do, it doesn&#8217;t matter one bit unless we&#8217;re treating it as an all-hands-on-deck emergency. We can&#8217;t keep launching products and pretending we&#8217;ll turn them into magical beautiful extensible platforms later. We&#8217;ve tried that and it&#8217;s not working.</p>
<p>The Golden Rule of Platforms, &#8220;Eat Your Own Dogfood&#8221;, can be rephrased as &#8220;Start with a Platform, and Then Use it for Everything.&#8221; You can&#8217;t just bolt it on later. Certainly not easily at any rate &#8212; ask anyone who worked on platformizing MS Office. Or anyone who worked on platformizing Amazon. If you delay it, it&#8217;ll be ten times as much work as just doing it correctly up front. You can&#8217;t cheat. You can&#8217;t have secret back doors for internal apps to get special priority access, not for ANY reason. You need to solve the hard problems up front.</p>
<p>I&#8217;m not saying it&#8217;s too late for us, but the longer we wait, the closer we get to being Too Late.</p>
<p>I honestly don&#8217;t know how to wrap this up. I&#8217;ve said pretty much everything I came here to say today. This post has been six years in the making. I&#8217;m sorry if I wasn&#8217;t gentle enough, or if I misrepresented some product or team or person, or if we&#8217;re actually doing LOTS of platform stuff and it just so happens that I and everyone I ever talk to has just never heard about it. I&#8217;m sorry.</p>
<p>But we&#8217;ve gotta start doing this right.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/steveys-google-platforms-rant/feed</wfw:commentRss>
		<slash:comments>1</slash:comments>
		</item>
		<item>
		<title>Competition of Android App Markets: Google, Amazon, Nook and more</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/competition-of-android-app-markets-google-amazon-nook-and-more</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/competition-of-android-app-markets-google-amazon-nook-and-more#comments</comments>
		<pubDate>Sat, 24 Sep 2011 06:16:20 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Android]]></category>
		<category><![CDATA[Fun & Life]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=556</guid>
		<description><![CDATA[
div.code {
  margin: 8px;
  margin-left: 32px;
  padding: 8px;
  background-color: #AAFFAA;
}


My educational math app A+ ITestYou is doing very well on the Google Android Market. Users signup and play daily. My next step is to publish the app to other markets and get more audience! For the next few weeks I will [...]]]></description>
			<content:encoded><![CDATA[<style>
div.code {
  margin: 8px;
  margin-left: 32px;
  padding: 8px;
  background-color: #AAFFAA;
}
</style>
<p><a href='https://market.android.com/details?id=com.vokamis.ityp'><img style='border: solid 2px #888888; width: 200px; margin: 12px; float: right;' src='http://www.softwaresecretweapons.com/jspwiki/attach/2011/09/320x480-9.png' /></a></p>
<p>My educational math app <a href='https://market.android.com/details?id=com.vokamis.ityp'>A+ ITestYou</a> is doing very well on the Google Android Market. Users signup and play daily. My next step is to publish the app to other markets and get more audience! For the next few weeks I will play with the Amazon and the Nook markets. Here you can find my notes.</p>
<h2>Google Android Market</h2>
<p>All Android developer tools, the device programming model, platform capabilities, phone emulators, Eclipse plugins are simply a pleasure to work with. You get step-by-step debugging and all the emulators including Galaxy Tab. <a href='http://developer.android.com/index.html'>Android Developers</a> is very easy to use and an application can be published in minutes. No approval required.</p>
<p>Be careful when uploading the app for the first time. The form is very long and there are traps awaiting! You are safe before you press the &#8220;publish&#8221; button. After that you can&#8217;t change the price or delete your apk files. Your app package name also becomes permanently assigned to this app, probably forever. I hope the Google team will figure out how to simplify the registration and clearly mark the write-once fields and the change-whenever-you-want fields.</p>
<p>Once uploaded, managing the app is very easy. You can have multiple APK files uploaded. Use the activate/deactivate buttons to choose between them. I especially like how it shows the exceptions that happened while the app runs on customer&#8217;s phone. Since we are in Java, it shows a complete stack trace of the exception! Simple in retrospect, but oh god it is useful. Proudly, I only have a single exception trace in the logs&#8230;</p>
<p>Another thing helped a lot &#8211; community help. Android platform had bugs and the documentation is not clear at times. But I was able to find help on the forums for every single issue I had. The http://www.stackoverflow.com was the best source.</p>
<p>App updates are still a mystery to me. We have three phones in the family, each is about 4 month old. Two are of the exact same model. All three have identical Android OS version and settings. Only one phone updates automatically, other two do not (yes, I have &#8220;Update Automatically&#8221; box is checked). the updates are visible on the phone in about 15-20 minutes after new APK is uploaded and activated, but automatic update is not triggered immediately after these updates are available. I still have to learn out how this all really works. More updates to code.</p>
<h2>Amazon Market</h2>
<p><a href='https://developer.amazon.com/home.html'>Amazon Developer Portal</a> requires application approval. It takes about 48 hours for a turn around. There is no message center where you can see your back and forth with the approver, only the most recent message is shown. But they send everything to you by email as well. APK update workflow is odd as well. When you edit the fields you may need to have re-approval, but you never know what change will trigger it. Why not warn the user that change blah and blah will require new approval?</p>
<p>During the review, I have received couple of very interesting objections from the approval team.</p>
<p><u>Objection 1:</u></p>
<div class="code">
One or more screen shots do not accurately reflect the application. Screen shots do not show advertising that appears in this app. Please submit screen shots that show advertising wherever it appears in this app.
</div>
<p>There was a specific questionnaire about ads, which I answered fully. The screenshots showed a selected portion of the screen, not the entire phone. Looks like it is not enough.</p>
<p><u>Objection 2:</u></p>
<div class="code">
We’ve noticed that A+ ITestYou Pro is $2.99 on the Amazon Appstore, but is a free download on a similar service.</div>
<p>True. I was in the process of registering both Free and Pro versions of the app in the Android Store. Pro app was not up yet, so it looked to a reviewer like the same app is being delivered over the Android Market for Free.</p>
<p>Think for a moment what this means. Shouldn&#8217;t I be able to charge whatever I want? Or not? Are we moving towards the DVD pricing model, where every movie starts out at $23.99 the first weeks it comes out and moves to $4.99 day by day? Same start price, regardless of how good the movie is. Sad&#8230; Or may be not&#8230; Lets start new <b>Luxury</b> Android Apps Market where we charge 10x of what other folks charge, except our shit will works 10x better!</p>
<p>One thing is not yet clear to me is how to test the Amazon market apps. There is no Amazon device yet, right? how do I get an app from Amazon app store to my phone? I still have to figure this out. My registration still pending. More updates to come.</p>
<h2>Nook Market</h2>
<p><a href='http://nookdeveloper.barnesandnoble.com/'>Nook Developer</a> folks were quick to turn around with the approval in just 12 hours. But there are issues.</p>
<p>Nook has its own device &#8211; Nook Color. The device emulator is not in the standard Android SDK (I was using SDK r12). Nook site has the instructions and getting the emulator was trivial. It plugs into the Android SDK manager quite nicely. So far so good.</p>
<p>But the emulator did not run very well, even after being updated to revision1. It crushed and did not support debugging. Threads were visible in Eclipse, but none of the breakpoints would fire. And the worst thing, my app crashed in the emulator all the time. In about 20 minutes I discovered that my app has nothing to do with it. While started in the emulator, the Nook browser crashes when it hits certain type of HTML pages, especially with JavaScript. And my app uses WebView in many places so it crashes too.</p>
<p>I logged a bug with the Nook support team. They have reproduced. Will see what happens next. More updates to come.</p>
<hr />
<p><i>DISCLAIMER: I am a full-time employee of Google Inc. at the time of writing. The opinions presented reflect my personal position and experience and are in no way associated with my work at Google Inc.</i></p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/competition-of-android-app-markets-google-amazon-nook-and-more/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Organization Domain Model strategic advantage</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/organization-domain-model-strategic-advantage</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/organization-domain-model-strategic-advantage#comments</comments>
		<pubDate>Tue, 30 Aug 2011 19:16:05 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Great People]]></category>
		<category><![CDATA[Meta-Modeling]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=518</guid>
		<description><![CDATA[Working at PayPal gave me the strong reasons to learn Domain Modeling. When you manage a large portfolio of systems, products and customers you need a concept of a “domain”. If you have the domains defined, you can now have variety of useful strategic conversations: 

How much funding does a domain need?
Who owns a domain?
Who [...]]]></description>
			<content:encoded><![CDATA[<p>Working at PayPal gave me the strong reasons to learn Domain Modeling. When you manage a large portfolio of systems, products and customers you need a concept of a “domain”. If you have the domains defined, you can now have variety of useful strategic conversations: </p>
<ul>
<li>How much funding does a domain need?</li>
<li>Who owns a domain?</li>
<li>Who is a decision making authority in a domain?</li>
<li>Should a domain be merged with another domain?</li>
<li>Should the feature be factored into a domain?</li>
</ul>
<p>If you don’t have domains, you end up managing tactically without strategy.</p>
<p>With my partners we spent endless hours modeling and debating…  What a domain is or isn’t? How big a feature should be to become a domain? How many domains is too many or not too many? Is domain an engineering or an organizational concept? Many difficult questions to answer. Most books focus on the systems and the engineering “domains”. But what about the products and the organizations? Shouldn’t these equally contribute into the domain model? Don’t we want the domains to facilitate organizational, product and technology alignment?</p>
<p>One sunny day this summer it all became clear as I read a brilliant <strong>Organization Domain Modeling</strong> guidebook by Mark Simos and Lockheed Martin Tactical Defense Systems (<a href='http://www.cs.bu.edu/fac/allemang/Papers/ODM.pdf'>original</a>). Just read the Chapter 3 “ODM Core Concepts”, which, in my opinion, is one of the best writings on the Enterprise Architecture and domain modeling. Not volumes of words, but a guidebook.</p>
<p>Being a Software Architect for over a decade now, I see how the role evolves. It is no longer about how to implement the systems, which can be outsourced or Google’d with ease. It is about defining the core principles, rules and constraints essential to the strategy of the organization and its competitiveness.  The common enabler is the Organization Domain Model.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/organization-domain-model-strategic-advantage/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Trying out Android SDK with WebView and AccountManager</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/trying-out-android-sdk-with-webview-and-accountmanager</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/trying-out-android-sdk-with-webview-and-accountmanager#comments</comments>
		<pubDate>Sun, 28 Aug 2011 06:00:57 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Android]]></category>
		<category><![CDATA[Fun & Life]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=524</guid>
		<description><![CDATA[
Android development is Easy!
It is 2AM now on Sunday&#8230; 6 hours ago I downloaded Android SDK for the first time. And now I have my own app running on the Nexus S phone! In just 6 hours, my educational math software arrived to the phone!
It was not easy&#8230; Had to learn the async programming model, [...]]]></description>
			<content:encoded><![CDATA[<p><a href='https://market.android.com/details?id=com.vokamis.ityp' target='_blank'><img border='0' style='width: 200px; float:right; margin-left: 32px; margin-bottom: 32px;' src='http://www.softwaresecretweapons.com/jspwiki/attach/2011/08/creative-phone.jpg' /></a></p>
<h2>Android development is Easy!</h2>
<p>It is 2AM now on Sunday&#8230; 6 hours ago I downloaded Android SDK for the first time. And now I have my own app running on the Nexus S phone! In just 6 hours, my <a href='http://www.itestyou.com'>educational math software</a> arrived to the phone!</p>
<p>It was not easy&#8230; Had to learn the async programming model, but doing Win32, Delphi and JavaScript for 10 years helped. Had to fight WebView, AccountManager for bugs and lack of documentation. Had to fight Eclipse, which hosed me couple of times. Had to fight the scope creep&#8230; But I did enjoy the capable tools with WYSIWYG UX and code complete all around! Enjoyed the emulator! Enjoyed the simplicity of deployment!</p>
<p>And I am done! All done in just 6 hours. With digital signing, and everything. Google Android Team and developer community ROCK! Phenomenal Platform! Great Success!</p>
<p>Let&#8217;s monetize it next weekend!!!</p>
<h2>WebView (android.webkit.WebView, API level 8, SDK r12)</h2>
<p>WebView is easy to use, but it has some minor issues if you want to control it&#8217;s behavior completely. It also has bugs&#8230; One bad one is in loadData() method, which does not handle raw HTML with some special characters. Here is how to escape raw HTML for use with with WebView:</p>
<div class="code">
<pre>

import android.webkit.WebView;

/**
 * WebView does not properly load HTML data with some special characters.
 * This method properly escapes HTML and loads the data into WebView.
 */
public static void loadData(WebView wv, String data, String mimeType, String encoding) {
	StringBuilder buf = new StringBuilder();
	int len = data.length();
	for (int i = 0; i < len; i++) {
		char chr = data.charAt(i);
		switch (chr) {
		case '%':
			buf.append("%25");
			break;
		case '\'':
			buf.append("%27");
			break;
		case '#':
			buf.append("%23");
			break;
		default:
			buf.append(chr);
		}
	}

	wv.loadData(buf.toString(), mimeType, encoding);
}
</pre>
</div>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/trying-out-android-sdk-with-webview-and-accountmanager/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Rationalization != Singularity</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/rationalization-singularity</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/rationalization-singularity#comments</comments>
		<pubDate>Thu, 18 Aug 2011 22:12:02 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=542</guid>
		<description><![CDATA[Rationalization != Singularity &#8211; I coined that!
It&#8217;s ok to rationalize things, but it is not ok to expect only one thing to win them all. Rationalization of 5 things into 3 is great! Can you push the remaining 3 into 1? Definitely not. Same principle applies to all artifacts at all levels of abstraction: state [...]]]></description>
			<content:encoded><![CDATA[<p><strong>Rationalization != Singularity</strong> &#8211; I coined that!</p>
<p>It&#8217;s ok to rationalize things, but it is not ok to expect only one thing to win them all. Rationalization of 5 things into 3 is great! Can you push the remaining 3 into 1? Definitely not. Same principle applies to all artifacts at all levels of abstraction: state variables, methods, classes, libraries, layers, services, databases, teams, development centers, business units, Vice Presidents, companies and nations.</p>
<p>Good luck in your rationalization!</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/rationalization-singularity/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Shooting for 99.9999999% reliability (yes NINE nines)</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/shooting-for-99-9999999-reliability-yes-nine-nines</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/shooting-for-99-9999999-reliability-yes-nine-nines#comments</comments>
		<pubDate>Fri, 29 Jul 2011 22:13:26 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Uncategorized]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/shooting-for-99-9999999-reliability-yes-nine-nines</guid>
		<description><![CDATA[How to achieve 99.9999999% reliability (yes NINE nines) in a software product with a 2 million lines of source code? Use message passing! Read here.
]]></description>
			<content:encoded><![CDATA[<p>How to achieve 99.9999999% reliability (yes NINE nines) in a software product with a 2 million lines of source code? Use message passing! Read <a href="http://jonasboner.com/2007/12/19/hotswap-code-using-scala-and-actors.html">here</a>.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/shooting-for-99-9999999-reliability-yes-nine-nines/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>The Trust Equation: modeling trust</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/the-trust-equation-modeling-trust</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/the-trust-equation-modeling-trust#comments</comments>
		<pubDate>Tue, 03 May 2011 07:37:31 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Meta-Modeling]]></category>
		<category><![CDATA[Smoke & Mirrors]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=462</guid>
		<description><![CDATA[It took me couple of year to build a trust with a specific person. And I think it ultimately failed. Why? I was not aware of The Trust Equation. 
The Trust is defined as a ratio of (Credibility + Reliability  + Intimacy) / Self-Interest. For any two people in the relationship there are two [...]]]></description>
			<content:encoded><![CDATA[<p>It took me couple of year to build a trust with a specific person. And I think it ultimately failed. Why? I was not aware of <a href='http://trustedadvisor.com/articles/the-trust-equation-a-primer'>The Trust Equation</a>. </p>
<p>The <strong>Trust</strong> is defined as a ratio of (<strong>Credibility</strong> + <strong>Reliability </strong> + <strong>Intimacy</strong>) / <strong>Self-Interest</strong>. For any two people in the relationship there are two Trust scores: trust of you in him and trust of him in you.     </p>
<p>Reflecting back I now realize that the Intimacy was the problem. It may not be too late to fix this after all…</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/the-trust-equation-modeling-trust/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Learning Advanced Computer Science on YouTube</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/learning-advanced-computer-science-on-youtube</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/learning-advanced-computer-science-on-youtube#comments</comments>
		<pubDate>Thu, 20 Jan 2011 22:44:28 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Smoke & Mirrors]]></category>
		<category><![CDATA[Text Mining]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=366</guid>
		<description><![CDATA[The Opportunity
I was very skeptical about online Universities, but this lecture series has changed my mind. Here I am waiting for my car oil change and I can refresh my understanding of data warehousing and data mining by watching a video on YouTube. Amazing!
If you get past the accent and the stuttering, this is a [...]]]></description>
			<content:encoded><![CDATA[<h2>The Opportunity</h2>
<p>I was very skeptical about online Universities, but this lecture series has changed my mind. Here I am waiting for my car oil change and I can refresh my understanding of data warehousing and data mining by watching a video on YouTube. Amazing!</p>
<p>If you get past the accent and the stuttering, this is a very good quality 300 series post Bachelor <a href='http://nptel.iitm.ac.in/video.php?courseId=1071&#038;v=WVSrsQ5Q94oF'>video course</a> by Dr. S. Srinath, India Institute of Technology (IIT), Madras. <a href="http://nptel.iitm.ac.in/index.php">IIT Madras</a> has many other courses, some as web pages, many in video format &#8211; all free. Why is not this being done in USA and Europe?</p>
<p>Watching a video does not replace reading books or attending actual University. But I somehow enjoy listening to a person describing the subject, especially for the material that I already know to some extent. Listening gives me time to reflect, synthesize and apply ideas to the problems in hand. The notes below are my own, made while watching. Enjoy!</p>
<h2>Lectures</h2>
<p><!--</p>
<h3>Lecture 1 &#8211; Introduction to Database Management System</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 1a &#8211; Conceptual Designs [53:40]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 2 &#8211; Conceptual Designs [54:00]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 3 &#8211; Relational Model [51:34]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 4 &#8211; Relational Model [54:09]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 5 &#8211; Structured Query Language [52:13]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 6 &#8211; Structured Query Language II [54:57]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 7 &#8211; ER Model to Relational Mapping [51:06]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 8 &#8211; Functional Dependencies and Normal Form [57:52]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 9 &#8211; ER Model to Relational Model Mapping [50:52]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 10 &#8211; Storage Structures [54:55]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 11 &#8211; Indexing Techniques Single Level [56:52]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 12 &#8211; Indexing Techniques Multi-Level [56:25]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 13 &#8211; Constraints and Triggers [57:30]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 14 &#8211; Query Processing and Optimization [56:41]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 15 &#8211; Query Processing and Optimization II [56:55]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 16 &#8211; Query Processing and Optimization &#8211; III [57:19]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 17 &#8211; Transaction Processing Concepts [55:31]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 18 &#8211; Transaction Processing and Database Manager [51:47]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 19 &#8211; Foundation for Concurrency Control [57:45]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 20 &#8211; Concurrency Control Part &#8211; I [58:25]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 21 &#8211; Concurrency Control Part &#8211; 2 [59:08]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 22 &#8211; Concurrency Control Part &#8211; 3 [57:48]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 23 &#8211; Concurrency Control Part &#8211; 4 [58:58]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 24 &#8211; Distributed Transaction Models [58:25]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 25 &#8211; Basic 2-Phase and 3-phase commit protocol [57:12]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 26 &#8211; Concurrency Control for Distributed Transaction [58:23]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 27 &#8211; Introduction to Transaction Recovery [55:23]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 28 &#8211; Recovery Mechanisms II [57:10]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 29 &#8211; Recovery Mechanisms III [55:43]</h3>
<ul>
<li></li>
</ul>
<hr />
<p>&#8211;></p>
<h3>Lecture 30 &#8211; Introduction to Data Warehousing and OLAP, Part I [57:49]</h3>
<ul>
<li>Operational efficiency (tactical) vs. strategic efficiency of databases: same underlying data, different nature of CRUD operations.</li>
<li>Data warehouse OLAP is a database system that is based on a join of different OLTP data sources (optimized for tactical operations), but optimized for the strategic decision making.</li>
<li>Transactional data (OLTP): requires frequent updates, has local relevance to a specific operation, accesses by point queries (queries that return one specific tuple: transaction, date, account, &#8230;), focus on reduced update time.</li>
<li>Warehouse data (OLAP): does not change the dataset, updated on predictable intervals, has huge amount of aggregation, global relevance, focus on reduced query time (O in OLAP means online, real time).</li>
<li>Data Warehouse: infrastructure to manage historical data, designed to support OLAP queries, trending, what-if and reporting.</li>
<li>Data Warehouse consists of several datamarts, each covering distinct OLTP entities. Data Mart is a collection of data about specific segment of the entire data warehouse. Data mart can support the same basic OLAP query and reporting as main data warehouse, but is focused on the specific segment of data.</li>
<li>Data cleaning and integration for OLAP in a two directional process, which produces data back flash into the OLTP system.</li>
<li>Data cleaning: Removes dirt, removes duplicates, enforces standardization &#038; semantics. Sources of data dirt: language/locale/encoding, abbreviations, semantic equivalence (Chennai == Madras), units of measure, missing values, use of 0/1 vs True/False vs T/F, address de-normalization, inconsistent values vs inconsistent relations, &#8230; There is no single cleaner can be uniformly used and it is difficult to automate the entire process. Lots of ad hock knowledge needs to be provided. Hard to validate and prone to GIGO (garbage in, garbage out) syndrome. Complexity increases with the number of data sources and over time.</li>
<li>Data cleaning steps are: analysis (finding data and metadata), definition of transformation rules, rule verification, transformation and backflow (replace original data with normalized representation). Data cleaning techniques. Hash-Merge duplicates elimination algorithm. Sorted neighborhood technique for misspelling detection and duplicate elimination; Graph-Based transitive closure algorithm to reduce the number of passes.</li>
<li>Integration: data and schema integration. Wrapper maps schema into metaschema and Extractor phisically esxtracts the data. Mediator maps individual metaschema into uber schema and Constructor joins the data together.</li>
<li>Tools: <a href='http://www.dataflux.com'>dfPower</a>,<a href='http://www.evtech.com'>ETI* Data Cleanse</a>, <a href='http://www.searchsoftware.co.uk'>SSA Name/Data Clustering Engine</a>.</li>
</ul>
<p><?php sswEmbedVideo('http://www.youtube-nocookie.com/v/m-aKj5ovDfg'); ?></p>
<hr />
<h3>Lecture 31 &#8211; Introduction to Data Warehousing and OLAP, Part II [58:11]</h3>
<ul>
<li>Data warehouse internals: involves aggregation, clustering, trending, multi-dimensional projections (how each dimension contributes to a total).</li>
<li>Typical warehouse has a hyper cube (data cube, which stores facts for dimensions) and materialized views (which store projections and aggregates). We do not store materialized views in the normal databases, but it is ok for OLAP as data is read-only. Hyper cubes support additional operators compared to classical SQL: pivoting (rotation of the hyper cube), slicing-dicing (extracting sub cubes), roll-up (aggregating across dimension) and drill-down (reveal details of an aggregated dimension).</li>
<li>Hyper cube implementations: ROLAP (multi-dimensional schema and queries are mapped into the relations model and is managed by a relational database; done in True Relational OLAP by Microstrategy) and MOLAP (native multi-dimensional model for storage and querying the data; done in EssBase).</li>
<li>Start schema architecture for OLAP cube. Snowflake schema is a star schema where dimension table has additional sub-dimensions in separate tables. Constellation model has multiple star schemas joined together, so there are multiple fact tables, but the dimension tables are shared.</li>
<li>Time dimension is special: found in most applications, has special meaning to the rollup (based on multiple calendars: tax year, academic or fiscal calendar), has interpretation for special events (release date, holiday, weekend), ordered dimension and order of traversal is very important. Revision history table.</li>
<li>Warehouse design starts with the enterprise model, then we build logical and physical model. Good warehouse design resembles enterprise model (not the operational model). Notion for ALL, to denote all dimention elements.</li>
<li>Classes of dimensions: categorical ({dog, cat}, {true, false}), ordinal ({1,2,3}, {Jan, Feb, &#8230;}), sparse (small number of unique values (T/F, Male/Female)), dense (large number of unique values). Types of indexes and storage: tabular (with multi-level sorting), space filling curve, mutli-dimensional indexes (ordered index on multiple attributes, partitioned hashing, grid files, bitmap indexes, KD trees, Quad trees, R trees for regions, signature trees and extensible hashing as indexes into categorical data).</li>
</ul>
<p><?php sswEmbedVideo('http://www.youtube-nocookie.com/v/G4NYQox4n2g'); ?></p>
<hr />
<h3>Lecture 32 &#8211; Case Study : MYSQL [57:55]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 33 &#8211; Case Study ORACLE and Microsoft Access [57:52]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 34 &#8211; Data Mining and Knowledge Discovery (Part 1) [54:46]</h3>
<ul>
<li>Value of data mining: extrapolation, wealth generation, security, strategic decision making, &#8230;</li>
<li>Statistical inference vs. data mining.</li>
<li>Interestingness: frequency, rarity, correlation, duration, consistency, periodicity, </li>
<li>Types of source data: tabular, spatial, temporal, tree, graph, sequence, text.</li>
<li>Support and confidence.</li>
<li>Apriori algorithm: frequent item-sets and association rules mining.</li>
</ul>
<p><?php sswEmbedVideo('http://www.youtube-nocookie.com/v/m5c27rQtD2E'); ?></p>
<hr />
<h3>Lecture 35 &#8211; Data Mining and Knowledge Discovery (Part 2) [57:59]</h3>
<ul>
<li>Classification (based on element difference) and clustering (based on element similarity)</li>
<li>Classification techniques: Hunt&#8217;s decision tree identification</li>
<li>Measures for similarity between elements projected into n-dimensional space</li>
<li>Clustering algorithms: nearest member (unknown number of clusters), iterative partitioning (number of clusters and their initial centers are known, we only reassigning members)</li>
<li>Sequence: finite ordered list of item-sets.</li>
<li>Mining sequence data: apriori algoritm for sequences</li>
<li>Language inference: finite state machine that produced the item-set sequence, general vs. specific inference with shortest-run generalization, </li>
<li>Mining streaming data: infinite sequence, no storage, real-time analysis.</li>
<li>Running and standing queries; unlike normal databases where data is standing, in streaming data query is standing and dataset is moving.</li>
<li>Running mean, running variance.</li>
<li>Gamma-consistency.</li>
</ul>
<p><?php sswEmbedVideo('http://www.youtube-nocookie.com/v/0hnqxIsXcy4'); ?></p>
<hr />
<h3>Lecture 36 &#8211; Object Oriented Databases [57:49]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 37 &#8211; Object Oriented Databases II [56:39]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 38 &#8211; XML &#8211; Introductory Concepts [57:51]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 39 &#8211; XML Advanced Concepts [57:40]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 40 &#8211; XML Databases [58:12]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 41 &#8211; Case Study &#8211; Part One Database Design [53:43]</h3>
<ul>
<li></li>
</ul>
<hr />
<h3>Lecture 42 &#8211; Case Study &#8211; Part 2 Database Design [58:36]</h3>
<ul>
<li></li>
</ul>
<hr />
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/learning-advanced-computer-science-on-youtube/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>You in the punchlines of all famous Web 2.0 companies</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/you-in-the-punchlines-of-all-famous-web-2-0-companies</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/you-in-the-punchlines-of-all-famous-web-2-0-companies#comments</comments>
		<pubDate>Sat, 27 Mar 2010 04:02:40 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Fun & Life]]></category>
		<category><![CDATA[Great People]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=182</guid>
		<description><![CDATA[Yes YOU are! Here are the punchlines of famous Web 2.0 companies. &#8220;You&#8221; is in almost everyone of them:


Social Networks:
Facebook: A social utility that connects you with people around you
MySpace: A place for friends


Entertainment:
YouTube: Broadcast Yourself
Weblo: Making earth virtually yours
Second Life: Your World. Your imagination.


Services:
Toronto.com: Be in the Know
Chickadvisor: Love it. Rate It. Share It.
Lavalife: [...]]]></description>
			<content:encoded><![CDATA[<p>Yes YOU are! Here are the punchlines of famous Web 2.0 companies. &#8220;You&#8221; is in almost everyone of them:</p>
<p />
<p />
<p><b>Social Networks:</b></p>
<p>Facebook: A social utility that connects you with people around you</p>
<p>MySpace: A place for friends</p>
<p />
<p />
<p><b>Entertainment:</b></p>
<p>YouTube: Broadcast Yourself</p>
<p>Weblo: Making earth virtually yours</p>
<p>Second Life: Your World. Your imagination.</p>
<p />
<p />
<p><b>Services:</b></p>
<p>Toronto.com: Be in the Know</p>
<p>Chickadvisor: Love it. Rate It. Share It.</p>
<p>Lavalife: Where singles click.</p>
<p />
<p />
<p><b>Games:</b></p>
<p>Zylom: Have fun</p>
<p>Shockwave: Where you’re free to play</p>
<p>WorldWinner: Its only a game. Until you play it.</p>
<p />
<p />
<p><b>Shopping:</b></p>
<p>Shopzilla.com: Discovery made simple</p>
<p>Edeals.com: Changing the way you shop online</p>
<p />
<p />
<p><b>Loyalty:</b></p>
<p>MyPoints.com: Rewarding smart shoppers.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/you-in-the-punchlines-of-all-famous-web-2-0-companies/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
		<item>
		<title>Splunk</title>
		<link>http://www.softwaresecretweapons.com/jspwiki/splunk</link>
		<comments>http://www.softwaresecretweapons.com/jspwiki/splunk#comments</comments>
		<pubDate>Fri, 19 Mar 2010 18:39:52 +0000</pubDate>
		<dc:creator>Pavel Simakov</dc:creator>
				<category><![CDATA[Smoke & Mirrors]]></category>

		<guid isPermaLink="false">http://www.softwaresecretweapons.com/jspwiki/?p=330</guid>
		<description><![CDATA[<b>Splunk</b> indexes data from any application, server or network device. It crawls logs, metrics, and other data from applications, servers and network devices and indexes it in a searchable repository from which it can generate graphs, reports and alerts. You can search and analyze billions of events across your software applications and IT infrastructure from one location in real time. It is easy to to troubleshoot your software application problems, crushes or investigate security incidents in minutes. You can monitor your live applications, set notifications and alerts to avoid outages and avoid service degradation. There are two editions Splunk Free and Splunk Enterprise. Free edition is good for managing small to medium web site.]]></description>
			<content:encoded><![CDATA[<p><a href="http://www.softwaresecretweapons.com/jspwiki/attach/2010/03/splunk_dashboard_800.png"><img src="http://www.softwaresecretweapons.com/jspwiki/attach/2010/03/splunk_dashboard_800-300x168.png" alt="" title="Splunk" width="300" height="168" class="size-medium wp-image-332" border='none' align='right' style='padding: 8px;'/></a></p>
<p>
<a href='http://www.splunk.com/'><b>Splunk</b></a> indexes data from any application, server or network device. It crawls logs, metrics, and other data from applications, servers and network devices and indexes it in a searchable repository from which it can generate graphs, reports and alerts. You can search and analyze billions of events across your software applications and IT infrastructure from one location in real time. It is easy to to troubleshoot your software application problems, crushes or investigate security incidents in minutes. You can monitor your live applications, set notifications and alerts to avoid outages and avoid service degradation. There are two editions Splunk Free and Splunk Enterprise. Free edition is good for managing small to medium web site.
</p>
<p>
<b>Index Live Data.</b> The more data you index, the more you&#8217;ll unlock about your IT infrastructure. Splunk Universal Indexing connects with every data source, including logs, configurations, traps and alerts, change events, the output of diagnostic commands (virtual and non-virtual), data from APIs and message queues, even multi-line logs from custom applications. Splunk indexing works without the need for specific parsers or adapters to write, maintain or buy. Once indexed, the same set of IT data is made available for troubleshooting, security incident investigations, network monitoring, compliance reporting and other valuables uses.
</p>
<p>
<b>Search and Investigate.</b> Once indexed, you can search for any event in your IT data. Don&#8217;t know what you&#8217;re looking for? Just start typing and Splunk&#8217;s Search Assistant will offer typeahead suggestions based on what’s in your data. You’ll also see suggested searches based on your search history and contextual help so that you can leverage the full power of Splunk’s search language.
</p>
<p>
<b>Modules.</b>Every element you see on a page in Splunk Web is a module, from the search bar to the results. Modules can be customized in XML. Here is customization example for the switcher module (notice expression in red):</p>
<div class="code">
<pre>

&lt;module name="HiddenSearch" group="eps Indexed over time" autoRun="True"&gt;
  &lt;param name="search"&gt;
    <span style="color:red;">index=_internal source=*metrics.log Component=metrics group=per_sourcetype_thruput | timechart avg(eps) by series</span>
  &lt;/param&gt;
  &lt;param name="earliest"&gt;-1h&lt;/param&gt;
    &lt;module name="StaticContentSample"&gt;
      &lt;param name="text"&gt;On this one I've thrown in some static text to describe the elements.&lt;/param&gt;
    &lt;/module&gt;
    &lt;module name="HiddenChartFormatter"&gt;
      &lt;param name="chart"&gt;line&lt;/param&gt;
      &lt;param name="primaryAxisTitle.text"&gt;Sourcetype&lt;/param&gt;
      &lt;param name="secondaryAxisTitle.text"&gt;events per second&lt;/param&gt;
      &lt;param name="legend.placement"&gt;right&lt;/param&gt;
        &lt;module name="JobProgressIndicator"/&gt;
        &lt;module name="FlashChart"&gt;
          &lt;param name="width"&gt;100%&lt;/param&gt;
          &lt;param name="height"&gt;300px&lt;/param&gt;
        &lt;/module&gt;
     &lt;/module&gt;
&lt;/module&gt;
</pre>
</div>
<p>
<b>Interact with Search Results.</b> Splunk also lets you interact with your search results. Zoom in and out on a timeline of your results to quickly reveal trends, spikes and anomalies. Click to drill down into your results and eliminate noise to get to the needle in the haystack. Whether you&#8217;re troubleshooting a ticket or investigating a security alert, you&#8217;ll get to the answer in seconds or minutes rather than hours or days and avoid needing to escalate issues to other groups to view the data you need.
</p>
<p>
<b>Custom Dashboards and Views.</b> Create custom dashboards in minutes with Splunk’s Dashboard Editor and turn your IT data into powerful insight. Dashboards let you organize your information for the needs of different users &#8211; technical and non-technical. Integrate charts, search results and even data from external applications. Build entirely personalized dashboards for management, security analysts, auditors, developers and sysadmins.
</p>
<p>
<b>Add Knowledge.</b> Splunk automatically extracts knowledge from your IT data to help you harness that information. You can add knowledge about events, fields, transactions, patterns and statistics on-the-fly to further enrich IT data and make the system smarter for all users by identifying, naming and tagging fields and datapoints. You can even add information from external asset management databases, configuration management systems and user directories.
</p>
<p>
<b>Monitor and Alert.</b> Any search can be saved and scheduled for continual monitoring and can trigger alerts via email or RSS. You can even kick-off a script to take remedial actions, send an SNMP trap to your system management console or generate a ticket at a service desk. Alerts can be triggered based on a variety of threshold, trend-based conditions and even more complex searches.
</p>
<p>
<b>Report and Analyze.</b> Splunk’s Report Builder helps you easily build advanced graphs and charts, visualize important trends, see highs and lows, summarize top values and report on the frequency of conditions. You can create robust, information-rich reports from scratch without an advanced knowledge of search commands. You can also save reports, integrate them into dashboards and share them with management or other colleagues in secure, read-only formats such as PDF.
</p>
<p>
<b>Build and Deploy IT Apps.</b> Build, package and deploy your own Apps on top of Splunk. The Splunk App Framework enables Customers and Partners to create their own Apps. Deliver a user experience tailored to a specific use case or augment existing vendor technologies. Once built, apply role-based access controls and deploy Apps with a tailored installation experience.
</p>
<p>
<b>Manage Splunk.</b> The Splunk Manager web interface is designed for users and administrators. Users can manage their own objects &#8211; saved searches, reports, event types and dashboards. Administrators can manage the overall installation, system configuration and security access rights. Splunk Deployment Server lets you centrally manage distributed Splunk deployments, which may include multiple Splunk Search Heads, Indexers and Forwarders across machines, datacenters and geographies.</p>
]]></content:encoded>
			<wfw:commentRss>http://www.softwaresecretweapons.com/jspwiki/splunk/feed</wfw:commentRss>
		<slash:comments>0</slash:comments>
		</item>
	</channel>
</rss>

