Amped::Technology

Powershell Baby Steps

Mon, 27 Oct 2008 16:06:54 GMT

Here are some representative tasks I've accomplished with Powershell:

Migrated filesystem+rdbms-based content into Sharepoint ( using sharepoint SDK )
Indexed SQL database using Lucene.NET. Step 2 is search from web page to produce more-like-this results for e-commerce web site, but the index powershell script will be scheduled to run nightly
synchronized data between source XML feed and target Sharepoint instance ( using sharepoint SOAP api )
used google geocoder to geoencode quickbase dataset ( downloaded as CSV). Wrote back results to Quickbase via REST api
wrote IIS log-tailing application with powershell+logparser to write latest log entries to sql server, where they're processed asynchronously by sql server service broker if the url's invoked require further action. Kind of a poor-man's apache module ...

Most of the above scripts used log4net for debugging. Example

$Log4NetHome = (pwd).path; $logpattern = "%date [%thread] %-5level [%x] - %message%newline"; # load the log4net library [void][Reflection.Assembly]::LoadFile($Log4NetHome + "\log4net.dll"); # configure logging [log4net.LogManager]::ResetConfiguration(); $Appender = new-object log4net.Appender.ConsoleAppender( new-object log4net.Layout.PatternLayout($logpattern)); [log4net.Config.BasicConfigurator]::Configure($Appender); $layout = new-object log4net.Layout.PatternLayout($logpattern); $Appender2 = new-object log4net.Appender.FileAppender($layout, "myfile.log"); [log4net.Config.BasicConfigurator]::Configure( $Appender2 ); # determines the log statements that show up $Appender2.Threshold = [log4net.Core.Level]::Info; $Appender.Threshold = [log4net.Core.Level]::Info; $Log = [log4net.LogManager]::GetLogger($Host.GetType()); $Log.Info("Started the main script..."); I feel like I know very little about Powershell, not fully grokking functions/cmdlets/pipeline vs iterators. The reason for this is that I get a lot done by dabbling, and don't have an excuse to go much further than the "copying 'net snippets" level of language mastery. The above is probably ~ 5 days of coding, spread out over 5 months. It's pretty much displaced perl as my data munging tool of choice.

Why? The integration with other tools I use is too good, no CPAN-on-Windows cognitive dissonance, and I can deploy it to client servers guilt-free since it's supported/blessed by MS, leaving me more likely to be off the critical path of support.

Powershell has a big advantage over similar efforts like groovy or boo, in that it's blessed by the platform vendor and thus destined for criticial-mass in mindshare, community, and google-footprint. And having it's own distinct google footprint means it doesn't break google, unlike jruby|jython|iron-whatever, etc. Code samples are easy to find, and you don't have to worry about the namespace/types clash inherent in running language a on runtime b.

Yep, I'm a fanboy.

SQL Server @ Amazon

Fri, 24 Oct 2008 13:50:17 GMT

Microsoft will be announcing their cloud offering next week at their PDC conference. When I first heard that Amazon had released their Windows-on-EC2 offering, I was thinking that the timing meant they were trying to steal some thunder from Microsoft. (Mary Jo agreed) When I saw the pricing, I wasn't so sure. SQL Server on EC2's smallest server is $1.10/hr, $26/day, $636/mo, $7 632/yr. So hosting an ASP.NET+SQL Server site on EC2 that goes over the 4Gig limit storage limit of free SQL Server Express will cost $8500/year if you have a SQL Server and an App Server. Or if you just want nothing-fancy ASP.NET hosting you could go with a shared server/shared database host and it'll cost $10/mo or $120/year. Hmmm. I know these alternatives aren't comparable, but it seems like two or three of the cheap accounts might get you similar reliability to the host-your-own option for a lot less dough. ( see mosso.com for an example of scale out vs scale up) This implies two things:

There's lots of room for a ISV to offer shared SQL Server hosting on EC2 and spread the SQL Server license over more customers and bring the price way down. (Elastra, XKoto ...)

It seems like this disparity gives Microsoft plenty of room to target their cloud offering price-wise. In other words, Amazon looks like they didn't get much more than a retail licensing deal out of Microsoft. What will MS do with their own zero-priced licencing? As I see it, MS has two competitive bullseyes to hit:

something like AppEngine, where you can run your ASP.NET apps backed by e.g. SSDS for free, and

something like EC2, where you have more workload flexibility but you don't need a whole enterprise OS and you don't want to get killed by licensing. They have some tough choices to make - the more they make use of their sweetheart deal on licensing costs, the more they stomp on their partners. I'm anxious to see how the stomping plays out next week.

The Black Swan: Living in a power-law world

Mon, 06 Oct 2008 13:09:00 GMT

Taleb's "The Black Swan" is a recently read book that describes in great detail one of the underlying phenomenom of the current credit crisis (via Gregor):

Our brains are wired for narrative, not statistical uncertainty. And so we tell ourselves simple stories to explain complex thing we don't--and, most importantly, can't--know. The truth is that we have no idea why stock markets go up or down on any given day, and whatever reason we give is sure to be grossly simplified, if not flat out wrong.
Nassim Nicholas Taleb first made this argument in Fooled by Randomness, an engaging look at the history and reasons for our predilection for self-deception when it comes to statistics. Now, in The Black Swan: the Impact of the Highly Improbable, he focuses on that most dismal of sciences, predicting the future. Forecasting is not just at the heart of Wall Street, but it’s something each of us does every time we make an insurance payment or strap on a seat belt.
The problem, Nassim explains, is that we place too much weight on the odds that past events will repeat (diligently trying to follow the path of the "millionaire next door," when unrepeatable chance is a better explanation). Instead, the really important events are rare and unpredictable. He calls them Black Swans, which is a reference to a 17th century philosophical thought experiment. In Europe all anyone had ever seen were white swans; indeed, "all swans are white" had long been used as the standard example of a scientific truth. So what was the chance of seeing a black one? Impossible to calculate, or at least they were until 1697, when explorers found Cygnus atratus in Australia.

(from the amazon page) I believe we learned everything we needed to prepare for the current crisis with the Long Term Capital Mgmt failure (risk model failure during market-wide sell-off; fund too big to fail thanks to positions in complicated unregulated derivates, failing; govt bailout) which the book discusses at length. And we have addressed none of the risks behind network failures in the intervening years, as interconnections have grown.

Here's hoping that the fix (and I don't mean the bailout) addresses systemic issues this time around and we don't just bury our head in the sand like we did last time, pretending to deal/regulate 1st order relationships while ignoring 2nd order+ effects of the financial web.

Update 1:

a better quote from the actual book...

"Globalization creates interlocking fragility, while reducing volatility and giving the appearance of stability. In other words it creates devastating Black Swans. We have never lived before under the threat of a global collapse. Financial Institutions have been merging into a smaller number of very large banks. Almost all banks are interrelated. So the financial ecology is swelling into gigantic, incestuous, bureaucratic banks – when one fails, they all fall. The increased concentration among banks seems to have the effect of making financial crisis less likely, but when they happen they are more global in scale and hit us very hard. We have moved from a diversified ecology of small banks, with varied lending policies, to a more homogeneous framework of firms that all resemble one another. True, we now have fewer failures, but when they occur ….I shiver at the thought."
— Nassim Taleb, The Black Swan (2006)

A pretty geeky article summarizing the book by the author

A funny-in-hindsight Forbes article from 7/2007 questioning the premise of the book:

Q:Is A Black Swan In The Way Of The 14,000 Dow?
A: Yes. Yes it is.

Loving Google Desktop

Mon, 06 Oct 2008 12:49:50 GMT

I really love the quicksearch function in the latest google desktop:

"The fastest way to search the web, and your own computer, is to use the Quick Search Box in the center of your desktop (simply press the "Ctrl" button twice to call up the search box, and press "Ctrl" twice again to hide it). Just type a few letters or words into the search box and your top results pop up instantly. You can also use it to launch applications without having to surf the Start menu; for example, you can launch Microsoft Word by typing "wor" into the Quick Search Box and selecting "Microsoft Word" in the list of results that appears. Or, simply type in your search term and press "Enter" to search the web."

I realize app-launchers is not a new thing, but I stay away from utilities like that because I switch desktops too often : it's annoying to rely too much on any customizations or desktop tweaks and few stand the test of time. Google desktop's search-web+gmail+desktop at the same time is too damn useful, but letting my launch apps speeds things up a lot.

[WindowsKey][WindowsKey]"CPAN" -- install new perl module using Strawberry perl cpan shell

[WindowsKey][WindowsKey]"Quicken" -- enter check from client etc.

Jonathan Ellis on AppEngine: Meh

Fri, 05 Sep 2008 11:27:53 GMT

Jonathan Ellis writes something suspiciously similar to what I think I would write about App Engine (but he actually went through the trouble of developing with it):

Having been eyeball deep in App Engine for a while, I've reluctantly concluded that I don't like it. I want to like it, since it's a great poster child for Python. And there are some bright spots, like the dirt-simple integration with google accounts. But it's so very very primitive in so many ways. Not just the missing features, or the "you can use any web framework you like, as long as it's django" attitude, but primarily a lot of the existing API is just so very primitive.

I can't get past the fact that there's no SQL. I know why there's no SQL, (Brewer's Conjecture etc) and I have some experience with it. I've been doing years of mostly-simple mashup stuff with quickbase, which is a similar PaaS-backing non-relational data store. It has it's place, but it's hard to imagine really cranking out something complicated with it. When I do code reviews for clients, the biggest improvements come from transforming procedural cursor logic back into set-oriented db-logic, and AppEngine wants you to go the opposite direction. Sigh.

Looking at the bigger picture, I'm trying to figure out what the drivers for PaaS adoption will be, especially in the case where the flavor of PaaS requires retiring your db muscles. Is giving the ability for developers (or non-developers) to clone and enhance code without sweating enough? Maybe there's a non-trivial demand for this (suggested by IBM's Project Zero, or stretching it a bit, MS-Sharepoint), but it seems like a long-term bet.

Aside: a cool App Engine project is Ben Adida's open source, cryptographically secure, online voting project Helios. You can conduct auditable elections and make sure they election-holders counted your vote the way you wanted. The auditing can be done by you personally, or a group that you trust. This all happens while hiding the actual candidate or choice you voted for, so you can't be coerced or sell it. Go Ben! The world needs Helios (but I'd be happy to start with just Florida).

Facebook Scaling

Fri, 22 Aug 2008 13:31:48 GMT

From Kellan:

Interesting to me was noting that:
they just got around to this 8 months ago, and they're fscking Facebook (which means you can wait)
they're still doing all writes to a single datacenter
they're hacking an object-level mark/sweep into the MySQL replication stream suggesting a certain parable of a hammer and nails.

Lol. Takeaway #3 is my favorite. Regarding the other two takeaways, I think Moore's Law will ensure the simplicity of single-datacenter and/or onsite deployments will be a compelling alternative to the clouds for quite some time. In other words, use Google App Engine et al if you like the cost and API's. For the distributed scalability, YNGNI.

The Paradox of Choice

Tue, 24 Jun 2008 10:38:45 GMT

The Paradox of Choice is a book with a thesis that's stuck in my head since I read about it a few years back. Simply put, more choice doesn't necessarily make us happier. The video and book gives some hard-to-dispute examples and data to back this up.

The thought that chases me is how much of technological progress is done in pursuit of choice/options/etc, without, as the book makes clear, accounting for the aggregate tragedy-of-the-commons cost of these options. The paradox is that someone somewhere will benefit (immensely?) from the choices technology produces, and there's no easy way to say when enough is enough.

Should we go back to 4-5 channels of TV? Laughable, right? Is it possible we'd be happier if we did? Hmmmm. That's just one way the book starts messing with your head.

The book repeats the central idea many times ... it's a simple enough that you get it well before the end, but I still found it worthwhile to ponder for the duration of the book. I would suggest that you pick up a copy, but that would only be adding to your reading choices (see how evil it is?)

Quick introduction on 'paradox' by author:

[via Stefano]

Roll Your Own Netflix

Mon, 23 Jun 2008 20:46:58 GMT

more praise for 'discontinuous fractional ownership'

In response to Props to Bookmooch, Replicate Technologies founder Ken Novak writes:

Hey John, this sounds good. For dvds, cds and paperbacks, zunafish.com works well (albeit not quite so karmic). For $1 plus $2 postage, the site sets up swaps. I own about 20 dvds, and I gradually swap them after I watch them, so that about 3/4 are ones I haven't seen. It's like netflix, with a bit less choice online but more DVDs at home and no fixed fees. Now if I could only acquire the time to watch everything I'd like to watch... #

I like the idea ... after getting over the initial Netflix rush during which the near-infinite film choice collided with finite movie watching time, I opted out and tried dvdswap.com. The experiment didn't last long, mainly because I had a big problem with liquidity. At the time, there just weren't enough swaps to be had. Now I just buy on half.com and resell on ebay. The buy/sell spread is usually about $4/movie (includes postage), which is the price I pay at my local video store, except no late fees, costs you nothing if you can't get to it, and no monthly commitment. I use Half.com because bidding on ebay is too much work and the selection of used inventory is pretty good.

Anyway, Bookmooch is incredibly liquid - I think the inventory is > 2 Million books and growing very rapidly. I think 'bigger than amazon' is definitely a possibility.