Early Adoptions, Health Checks and New Year Rants.

Generations

Two days ago I read the following Tweet by Hugh MacLeod:

"[…] Early Adopter Problem: How to differentiate from the bandwagon, once the bandwagon starts moving faster than you are […]"

That makes me think of early adoption of a few technologies I have been working with, and how the community around those evolved. For example:

Operations Manager… early adoption meant that I have been working with it since the beta, had posted one of the earliest posts about how to use a script in a Unit Monitor back in may 2007 (the product was released in April 2007 and there was NO documentation back then, so we had to really try to figure out everything…), but someone seems to think it is worth repeating the very same lesson in November 2008, with not a lot of changes, as I wrote here. I don't mean being rude to Anders… repeating things will surely help the late adopters finding the information they need, of course.

Also, I started playing early with Powershell. I posted my first (and only) cmdlet back in 2006. It was not a lot more than a test for myself to learn how to write one, but that's just to say that I started playing early with it. I have been using it to automate tasks for example.

Going back to the quote above, everyone gets on the bandwagon posting examples and articles. I had been asked a few times about writing articles on OpsMgr and Powershell usage (for example by www.powershell.it) but I declined, as I was too busy using this knowledge to do stuff for work (where “work” is defined as in “work that pays your mortgage”), rather than seeking personal prestige through articles and blogs. Anyway, that kind of articles are appearing now all over the Internet and the blogosphere now. The above examples made me think of early adoption, and the bandwagon that follows later on… but even as an early adopter, I was never very noisy or visible.

Now, going back to what I do for work, (which I mentioned here and here in the past), I work in the Premier Field Engineering organization of Microsoft Services, which provides Premier services to customers. Microsoft Premier customer have a wide range of Premier agreement features and components that they can use to support their people, improve their processes, and improve the productive use of the Microsoft technology they have purchased. Some of these services we provide are known to the world as “Health Checks”, some as “Risk Assessment Programs” (or, shortly, RAPs). These are basically services where one of our technology experts goes on the customer site and there he uses a custom, private Microsoft tool to gather a huge amount of data from the product we mean to look at (be it SQL, Exchange, AD or anything else….). The Health Check or RAP tool collects the data and outputs a draft of the report that will be delivered to the customer later on, with all the right sections and chapters. This is done so that every report of the same kind will look consistent, even if the engagement is performed by a different engineer in a different part of the world. The engineer will of course analyze the collected data and write recommendations about what is configured properly and/or about what could or should be changed and/or improved in the implementation to make it adhere to Best Practices. To make sure only the right people actually go onsite to do this job we have a strict internal accreditation process that must be followed; only accredited resources that know the product well enough and know exactly how to interpret the data that the tool collects are allowed to use it and to deliver the engagement, and present/write the findings to the customer.

So why am I telling you this here, and how have I been using my early knowledge of OpsMgr and Powershell for ?

I have used that to write the Operations Manager Health Check, of course!

We had a MOM 2005 Health Check already, but since the technology has changed so much, from MOM to OpsMgr, we had to write a completely new tool. Jeff  (the original MOM2005 author, who does not have a blog that I can link to) and me are the main coders of this tool… and the tool itself is A POWERSHELL script. A longish one, of course (7000 lines, more or less), but nothing more than a Powershell script, at the end of the day. There are a few more colleagues that helped shape the features and tested the tool, including Kevin Holman. Some of the database queries on Kevin’s blog are in fact what we use to extract some of the data (beware that some of those queries have recently been updated, in case you saved them and using your local copy!), while some other information are using internal and/or custom queries. Some other times we use OpsMgr cmdlets or go to the SDK service, but a lot of times we query the database directly (we really should use the SDK all the times, but for certain stuff direct database access is way faster). It took most of the past year to write it, test it, troubleshoot it, fix it, and deliver the first engagements as “beta” to some customers to help iron out the process… and now the delivery is available! If a year seems like a long time, you have to consider this is all work that gets done next to what we all have to normally do with customers, not replacing it (i.e. I am not free to sit on my butt all day and just write the tool… I still have to deliver services to customers day in day out, in the meantime).

Occasionally, during this past calendar year, that is approaching its end, I have been willing and have found some extra time to disclose some bits and pieces, techniques and prototypes of how to use Powershell and OpsMgr together, such as innovative ways to use Powershell in OpsMgr against beta features, but in general most of my early adopter’s investment went into the private tool for this engagement, and that is one of the reasons I couldn’t blog or write much about it, being it Microsoft Intellectual Property.

But it is also true that I did not care to write other stuff when I considered it too easy or it could be found in the documentation. I like writing of ideas, thoughts, rants OR things that I discover and that are not well documented at the time I study them… so when I figure out things I might like leaving a trail for some to follow. But I am not here to spoon feed people like some in the bandwagon are doing. Now the bandwagon is busy blogging and writing continuously about some aspect of OpsMgr (known or unknown, documented or not), and the answer to the original question of Hugh is, in my opinion, that it does not really matter what the bandwagon is doing right now. I was never here to do the same thing. I think that is my differentiator. I am not saying that what a bunch of colleagues and enthusiasts is doing is not useful: blogging and writing about various things they experiment with is interesting and it will be useful to people. But blogs are useful until a certain limit. I think that blogs are best suited for conversations and thoughts (rather than for "howto's"), and what I would love to see instead is: less marketing hype when new versions are announced and more real, official documentation.

But I think I should stop caring about what the bandwagon is doing, because that's just another ego trip at the end of the day. What I should more sensibly do, would be listening to my horoscope instead:

[…] "How do you slay the dragon?" journalist Bill Moyers asked mythologist Joseph Campbell in an interview. By "dragon," he was referring to the dangerous beast that symbolizes the most unripe and uncontrollable part of each of our lives. In reply to Moyers, Campbell didn't suggest that you become a master warrior, nor did he recommend that you cultivate high levels of sleek, savage anger. "Follow your bliss," he said simply. Personally, I don't know if that's enough to slay the dragon — I'm inclined to believe that you also have to take some defensive measures — but it's definitely worth an extended experiment. Would you consider trying that in 2009? […]

Programmatically Check for Management Pack updates in OpsMgr 2007 R2

One of the cool new features of System Center Operations Manager 2007 R2 is the possibility to check and update Management Packs from the catalog on the Internet directly from the Operators Console:

Select Management Packs from Catalog

Even if the backend for this feature is not yet documented, I was extremely curious to see how this had actually been implemented. Especially since it took a while to have this feature available for OpsMgr, I had the suspicion that it could not be as simple as one downloadable XML file, like the old MOM2005's MPNotifier had been using in the past.

Therefore I observed the console's traffic through the lens of my proxy, and got my answer:

ISA Server Log

So that was it: a .Net Web Service.

I tried to ask the web service itself for discovery information, but failed:

WSDL

Since there is no WSDL available, but I badly wanted to interact with it, I had to figure out: what kind of requests would be allowed to it, how should they be written, what methods could they call and what parameters should I pass in the call. In order to get started on this, I thought I could just observe its network traffic. And so I did… I fired up Network Monitor and captured the traffic:

Microsoft Network Monitor 3.2

Microsoft Network Monitor is beautiful and useful for this kind of stuff, as it lets you easily identify which application a given stream of traffic belongs to, just like in the picture above. After I had isolated just the traffic from the Operations Console, I then saved those captures packets in CAP format and opened it again in Wireshark for a different kind of analysis – "Follow TCP Stream":

Wireshark: Follow TCP Stream

This showed me the reassembled conversation, and what kind of request was actually done to the Web Service. That was the information I needed.

Ready to rock at this point, I came up with this Powershell script (to be run in OpsMgr Command Shell) that will:

1) connect to the web service and retrieve the complete MP list for R2 (this part is also useful on its own, as it shows how to interact with a SOAP web service in Powershell, invoking a method of the web service by issuing a specially crafted POST request. To give due credit, for this part I first looked at this PERL code, which I then adapted and ported to Powershell);

2) loop through the results of the "Get-ManagementPack" opsmgr cmdlet and compare each MP found in the Management Group with those pulled from the catalog;

3) display a table of all imported MPs with both the version imported in your Management Group AND the version available on the catalog:

Script output in OpsMgr Command Shell

Remember that this is just SAMPLE code, it is not meant to be used in production environment and it is worth mentioning again that OpsMgr2007 R2 this is BETA software at the time of writing, therefore this functionality (and its implementation) might change at any time, and the script will break. Also, at present, the MP Catalog web service still returns slightly older MP versions and it is not yet kept in sync and updated with MP Releases, but it will be ready and with complete/updated content by the time R2 gets released.

Disclaimer

The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my own personal opinion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.
THIS WORK IS NOT ENDORSED AND NOT EVEN CHECKED, AUTHORIZED, SCRUTINIZED NOR APPROVED BY MY EMPLOYER, AND IT ONLY REPRESENT SOMETHING WHICH I'VE DONE IN MY FREE TIME. NO GUARANTEE WHATSOEVER IS GIVEN ON THIS. THE AUTHOR SHALL NOT BE MADE RESPONSIBLE FOR ANY DAMAGE YOU MIGHT INCUR WHEN USING THIS INFORMATION. The solution presented here IS NOT SUPPORTED by Microsoft.

Backup or Store stuff to GMail via IMAP in Ruby

Once upon a time, I used to store some automated small backups into GMail just by having the scheduled backup send an email to my GMail account. At one stage they blocked me from doing so, marking those repeated email as SPAM.

After that, I took a different approach: I kept sending the mail on the SAME server as the backup, and using IMAP I could DRAG-and-DROP the backup attachment from the mailbox on one server to the mailbox on another server (=GMail). They did not mark me as a spammer that way, of course.
So that worked for a while, but then I got tired of doing this manually.

So the following ruby script is the way I automated the "move offsite" part of that backup.
For completeness, I will give the due credits about who set me on the right track: I started off by this example by Ryan.

#!/usr/bin/env ruby
begin_ = Time.now

#includes
require 'net/imap'

##Source Info
$SRCSERVER="mail.muscetta.com"
$SRCPORT=143
$SRCSSL=false
$SRCUSERNAME="daniele"
$SRCPASSWORD=""
$SRCFOLDER="INBOX.Backups"

##Destination Info
$DSTSERVER="imap.gmail.com"
$DSTPORT=993
$DSTSSL=true
$DSTUSERNAME="muscetta@gmail.com"
$DSTPASSWORD=""
$DSTFOLDER="Backup"

#connect to source
puts "connecting to source server #{$SRCSERVER}... nn"
srcimap = Net::IMAP.new($SRCSERVER,$SRCPORT,$SRCSSL)
srcimap.login($SRCUSERNAME, $SRCPASSWORD)
srcimap.select($SRCFOLDER)

#connect to destination
puts "connecting to destination server #{$DSTSERVER}... nn"
dstimap = Net::IMAP.new($DSTSERVER,$DSTPORT,$DSTSSL)
dstimap.login($DSTUSERNAME, $DSTPASSWORD)
dstimap.select($DSTFOLDER)

# Loop through all messages in the source folder.
uids = srcimap.uid_search(['ALL'])
if uids.length > 0
	$count = uids.length
	puts "found #{$count} messages to move... nn"

	srcimap.uid_fetch(uids, ['ENVELOPE']).each do |data|
		mid = data.attr['ENVELOPE'].message_id

		# Download the full message body from the source folder.
		puts "reading message... #{mid}"
		msg = srcimap.uid_fetch(data.attr['UID'], ['RFC822', 'FLAGS', 'INTERNALDATE']).first

		# Append the message to the destination folder, preserving flags and internal timestamp.
		puts "copying message #{mid} to destination..."
		dstimap.append($DSTFOLDER, msg.attr['RFC822'], msg.attr['FLAGS'], msg.attr['INTERNALDATE'])

		#delete the msg
		puts "deleting messsage #{mid}..."
		srcimap.uid_store(data.attr['UID'], '+FLAGS', [:Deleted])
		srcimap.expunge

	end

	#disconnect
	dstimap.close
	srcimap.close
end

total_time = Time.now - begin_
puts "Done. RunTime: #{total_time} sec. nn"

CentOS 5 Management Pack for OpsMgr SCX

As I mentioned here, I have been testing the SCX beta.

Not having one of the "supported" platforms pushed me into playing with the provided Management Packs, and in turn I managed to use the MP for Red Hat Enterprise Linux 5 as a base, and replaced a couple of strings in the discoveries in order to get a working CentOS 5 Management Pack.

CentOS_HealthExplorer01_NEW

I still have not looked into the "hardware" monitors and health model / service model, so those are not currently monitored. But it is a start.

A lot of people have asked me a lot of information and would like to get the file – both in the blog's comment, on the newsgroup, or via mail. I am sorry, but I cannot provide you with the file, because it has not been throughly tested and might render your systems unstable, and also because there might be licensing and copyright issues that I have not checked within Microsoft.

Keep also in mind that using CentOS as a monitored platform is NOT a SUPPORTED scenario/platform for SCX. I only used it because I did not have a Suse or Redhat handy that day, and because I wanted to understand how the Management Packs using WS-Man worked.

This said, should you wish to try to do the same "MP Hacking" I did,  I pretty much explained all you need to know in my previous post and its comments, so that should not be that difficult.

Actually, I still think that the best way to figure out how things are done is by looking at the actual implementation, so I encourage you to look at the management packs and figure out how those work. There are a few mature tools out there that will help you author/edit Management Packs if you don't want to edit the XML directly: the Authoring Console, and Silect MP Studio Lite, for example. If you want to delve in the XML details, instead, then I suggest you read the Authoring Guide and peek at Steve Wilson's AuthorMPs.com site.

Disclaimer
The information in this weblog is provided "AS IS" with no warranties, and confers no rights. This weblog does not represent the thoughts, intentions, plans or strategies of my employer. It is solely my own personal opinion. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.
THIS WORK IS NOT ENDORSED AND NOT EVEN CHECKED, AUTHORIZED, SCRUTINIZED NOR APPROVED BY MY EMPLOYER, AND IT ONLY REPRESENT SOMETHING WHICH I'VE DONE IN MY FREE TIME. NO GUARANTEE WHATSOEVER IS GIVEN ON THIS. THE AUTHOR SHALL NOT BE MADE RESPONSIBLE FOR ANY DAMAGE YOU MIGHT INCUR WHEN USING THIS PROGRAM.

Live ID WordPress Plugin

This is the page of the Windows Live ID Authentication WordPress Plugin.

I started messing about with Windows Live ID SDK and PHP in November 2007. My first post detailing the first version of this plugin is at http://www.muscetta.com/2007/11/02/using-live-id-to-authenticate-to-wordpress/

Changelog/History

  • v0.1 – The first version was very basic and let you authenticate, but that was it. It was necessary to manually edit the database, so it was just a little more than a proof of concept.
  • v0.2 – The second version was just a fix to let the plugin work with the new authentication secure cookies in WordPress 2.5. It did not work anymore in 2.3 and older, as I completely changed the code
  • v0.3c – this is the first version to include BOTH authentication codes (so it will detect which version of WordPress you are running and will use the correct authentication code). This has only been throughly tested on 2.5 anyway, so the rest should work, but I am not 100% sure about it. Also, this version finally includes a plugin configuration page, so you won't have to manually edit the database to associate your Live ID account with your WordPress account.
  • v0.3d – Added the possibility to REMOVE the association between Live ID and WordPress ID, once set, in case you got it wrong and you want to repeat the process, or in case you want to change the Live ID account to be associated with your WordPress acount.
  • v0.4 – Update for WordPress 2.7

Installation

  1. register your blog as an application like it is explained at http://msdn2.microsoft.com/en-us/library/bb676626.aspx
  2. you will get a key for your application.
  3. When registering your app with Live, the "Return URL" is going to be something like http://<your-blog-url>/wp-content/plugins/liveauth/webauth-handler.php
  4. You then will have to enter the same key/information you got from the registration process in the Application-Key.xml file. This file should be kept in a directory where it cannot be downloaded from the internet. Use something outside your webroot if you can. Otherwise just set restrictive permissions of the file so that it cannot be downloaded from the Internet. This is very important: access to your secret key in the config file will let an attacker decrypt your cookies and spoof your identity. This recomendation should not be overlooked. Once you selected a location for the Application-Key.xml file, point at this file by modifying the settings.php file.
  5. Activate the plugin.
  6. Go to http://<your-blog-url>/wp-content/plugins/liveauth/auth.php
  7. Log In with your Live ID. you should return to the same page, that will tell you you are logged in to Live but not to WordPress. Take note of your Live ID number from here (as in "select" and "copy" it).
  8. Click on the link that will let you authenticate with your existing WordPress account. You will be then redirected to the option page for the Live ID plugin.
  9. Type in or paste the ID you copied in step 7. Submit the form.
  10. You should be set. Now WordPress knows your Live ID corresponds to your WordPress account.

Disclaimer
The information in this weblog is provided "AS IS" with no warranties, and confers no rights. All code samples are provided "AS IS" without warranty of any kind, either express or implied, including but not limited to the implied warranties of merchantability and/or fitness for a particular purpose.
Even if I was working at Microsoft at the time I wrote this, THIS WORK WAS NOT ENDORSED AND NOT EVEN CHECKED, AUTHORIZED, SCRUTINIZED NOR APPROVED BY MICROSOFT, AND IT ONLY REPRESENT SOMETHING WHICH I DID IN MY FREE TIME. NO GUARANTEE WHATSOEVER IS GIVEN ON THIS. THE AUTHOR SHALL NOT BE MADE RESPONSIBLE FOR ANY DAMAGE YOU MIGHT INCUR WHEN USING THIS PROGRAM.

This code includes part of the Live ID SDK Sample PHP application, therefore it is subject to the same License and Terms of Service about the ways it can be used and redistributed. In plain english, those terms mean I can't make it GPL, and therefore I cannot add it to the official wordpress plugins gallery, albeit I have been using it for years and works just great – it is just a 'code sample' 😉

Popfly Virtual Earth Mashup on Moonlight

Popfly Virtual Earth Mashup on Moonlight

Installed moonlight on Ubuntu from source by following these instructions (there are some typo's but they are understandable and correctable).

All in all, even being still under heavy development, what Miguel de Icaza has achieved (with moonlight, just like with mono) is amazing.

After I posted the above picture on Flickr, John Montgomery was amazed to see PopFly (his creature) working on moonlight, and he linked to me from his blog.

Looking at OpsMgr2007 Alert trend with Command Shell

It's friday night, I am quite tired and I can't be asked of writing a long post. But I have not written much all week, not even updated my Twitter, and now I want to finish the week with at least some goodies. So this is the turn of a couple of Powershell commands/snippets/scripts that will count alerts and events generated each day: this information could help you understand the trends of events and alerts over time in a Management Group. It is nothing fancy at all, but they can still be useful to someone out there. In the past (MOM 2005) I used to gather this kind of information with SQL Queries against the operations database. But now, with Powershell, everything is exposed as objects and it is much easier to get information without really getting your hands dirty with the database 🙂

#Number of Alerts per day

$alerttimes = Get-Alert | Select-Object TimeRaised
$array=@()

foreach ($datetime in $alerttimes){
$array += $datetime.timeraised.date
}

$array | Group-Object Date

#Number of Events per day

$eventtimes = Get-Event | Select-Object TimeGenerated
$array=@()

foreach ($datetime in $eventtimes){
$array += $datetime.timegenerated.date
}

$array | Group-Object Date

Beware that these "queries" might take a long time to execute (especially the events one) depending on the amount of data and your retention policy.

This is of course just scratching the surface of the amount of amazing things you can do with Powershell in Operations Manager 2007. For this kind of information you might want to keep an eye on the official "System Center Operations Manager Command Shell" blog: http://blogs.msdn.com/scshell/

Get-FlickrPhotos

A while ago, talking to some friends, I was mentioning how cool it was that Flickr provides APIs, so that you can always get your data out of it, if you want to. There are several downloader applications that I found on the Internet, but I have not yet chosen one that I completey like among the few that I've tried. So, inspired by Kosso's PHP script for enumerating your photos on Flickr, I thought I'd port it to Powershell and make my own version of it. Just for the fun of it. My Powershell script does not do everything that Kosso's one does: I don't build a web page showing description and comments. I suppose this is because the original script was made with PHP, which you usually run on a web server and outputting as HTML is the standard thing you would do in PHP. I just concentrated on the "download" thing, since mine it is a console script. You can think of mine as a "full backup" script. Full… well, at least of all your photos, if not of all the metadata. It should be trivial to extend anyway, also considering Powershell XML type accelerator really makes it extremely easy to parse the output of a REST API such as Flickr's (I would say even easier and more readable that PHP'simplexml). There is a ton of things that could be extended/improved in the script… including supporting proxy servers, accepting more parameters for things that are now hardcoded… and with a million other things. Even this way, though, I think that the script can be useful to show a number of techniques in Powershell. Or just to download your photos 🙂 So you can download the script from here: Get-FlickrPhotos.ps1

Simply Works

Simply Works

Simply Works, uploaded by Daniele Muscetta on Flickr.

I don't know about other people, but I do get a lot to think when the end of the year approaches: all that I've done, what I have not yet done, what I would like to do, and so on…

And it is a period when memories surface.

I found the two old CD-ROMs you can see in the picture. And those are memories.
missioncritical software was the company that invented a lot of stuff that became Microsoft's products: for example ADMT and Operations Manager.

The black CD contains SeNTry, the "enterprise event manager", what later became Operations Manager.
On the back of the CD, the company motto at the time: "software that works simply and simply works".
So true. I might digress on this concept, but I won't do that right now.

I have already explained in my other blog what I do for work. Well, that was a couple of years ago anyway. Several things have changed, and we are moving towards offering services that are more measurable and professional. So, since it happens that in a certain job you need to be an "expert" and "specialize" in order to be "seen" or "noticed".
You know I don't really believe in specialization. I have written it all over the place. But you need to make other people happy as well and let them believe what they want, so when you "specialize" they are happier. No, really, it might make a difference in your carrer 🙂

In this regard, I did also mention my "meeting again" with Operations Manager.
That's where Operations manager helped me: it let me "specialize" in systems and applications management… a field where you need to know a bit of everything anyway: infrastructure, security, logging, scripting, databases, and so on… 🙂
This way, everyone wins.

Don't misunderstand me, this does not mean I want to know everything. One cannot possibly know everything, and the more I learn the more I believe I know nothing at all, to be honest. I don't know everything, so please don't ask me everything – I work with mainframes 🙂
While that can be a great excuse to avoid neighbours and relatives annoyances with their PCs though, on the serious side I still believe that any intelligent individual cannot be locked into doing a narrow thing and know only that one bit just because it is common thought that you have to act that way.

If I would stop where I have to stop I would be the standard "IT Pro". I would be fine, sure, but I would get bored soon. I would not learn anything. But I don't feel I am the standard "IT Pro". In fact, funnily enough, on some other blogs out there I have been referenced as a "Dev" (find it on your own, look at their blogrolls :-)). But I am not a Dev either then… I don't write code for work. I would love to, but I rarely actually do, other than some scripts. Anyway, I tend to escape the definition of the usual "expert" on something… mostly because I want to escape it. I don't see myself represented by those generalization.

As Phil puts it, when asked "Are software developers – engineers or artists?":

"[…] Don’t take this as a copout, but a little of both. I see it more as craftsmanship. Engineering relies on a lot of science. Much of it is demonstrably empirical and constrained by the laws of physics. Software is less constrained by physics as it is by the limits of the mind. […]"

Craftmanship. Not science.
And stop calling me an "engineer". I am not an engineer. I was even crap in math, in school!

Anyway, what does this all mean? In practical terms, it means that in the end, wether I want it or not, I do get considered an "expert" on MOM and OpsMgr… and that I will mostly work on those products for the next year too. But that is not bad, because, as I said, working on that product means working on many more things too. Also, I can point to different audiences: those believing in "experts" and those going beyond schemes. It also means that I will have to continue teaching a couple of scripting classes (both VBScript and PowerShell) that nobody else seems to be willing to do (because they are all *expert* in something narrow), and that I will still be hacking together my other stuff (my facebook apps, my wordpress theme and plugins, my server, etc) and even continue to have strong opinions in those other fields that I find interesting and where I am not considered an *expert* 😉

Well, I suppose I've been ranting enough for today…and for this year 🙂
I really want to wish everybody again a great beginning of 2008!!! What are you going to be busy with, in 2008 ?