Day-3 of pre #PuppetConf Training: Puppet Fundamentals for System Administrators #puppetize @PuppetConf

This is a reflection on the 3rd and final day of PuppetConf pre-conf training. The full conference took place on Thursday and Friday.  To see what we’ve done on Monday & Tuesday check out my previous posts on Day 1 and Day 2.

By the third day in class, we had built out enough scaffolding (both virtual and cerebral) to leverage additional features of puppet enterprise in a master/agent environment and further expand to, testing, deploying and analyzing the modules that we’d created in class as well as a few module from the community.

As an example of the in class exercise, we progressively developed and deployed an apache module to install the package(s) and dependencies and configure it to serve up several virtual hosts on a single server. We then used Puppet to further configure each individual virtual host to publish a dynamically generated index page containing content that pulled from the server information that facter knew about. This last part was accomplished via ‘erb’ templates, pulling in facter facts and some variables from our own classes.

We also listed, searched for, downloaded and installed modules from the Puppet Forge. This was a great experience because at this point in the class I now felt like I had a better working understanding of the structure and purpose of each of the components of a module.  Even though I still may not yet feel confident enough to author and publish my own puppet module, I do feel I can pick a part and evaluate an exisiting module that is provided by a community member who likely had much more experience using Puppet. Nothing like standing on the shoulders of giants to get started using a powerful tool. That’s the way I learn a lot of things, buy observing how others who have gone before me do it and then taking that knowledge and information and recombining it into something that meet my own specific purposes.

The balance of the remaining lecture and slides covered topics of class inheritance to share common behavior among multiple classes by inheriting parent scopes and how to override parameters should you prefer not to use the default values.

Hiera yaml was introduced as an external data lookup tool that can be used in your manifests with key:value pairs to obfuscate the actual variable values. I need to understand this particular tool better to get a feel for how it will be helpful. On the surface, it seems to be useful for publishing reusable manifests without revealing or hardcoding info from a particular environment.

Live management is a cool feature of the puppet enterprise console that we went over in class. Live Management can be used to inspect resources across all nodes managed by a Puppet Enterprise server. This is a powerful tool that provides the ability to inquire live across an entire environment – say check for a package or user, where they are and how they are config’d including the number of any variations. Live management can also be leveraged to instantiate changes live on managed systems. After covering the concept and seeing a few demonstrations of the GUI console in the lecture, we started a Live Management lab using our classroom infrastructure.  This lab exercise was a bit ‘character building’.  It required that all 15 of the students nodes connect up to the instructor’s master server and then ask that server to turn around and live query all of the managed agent nodes (that’s 15 agents each querying 15 agents… from a single training VM  (2GB RAM, 2 vCPU) on the instructor’s laptop. Obviously an under-provisioned VM running on the modestly equipped instructor laptop was not representative of a production deployment and it was felt. Things timed out… the task queues filled up, the UI became unresponsive… students scratched their heads, instructors sighed and grimaced. This was pushing the envelope. Approaching the latter part of the final day of the course, and this was the very first exercise/demo that had gone even a bit awry. As technologist, everyone in the class understood what was going on, the reason for it and what we were supposed to get out of doing the exercise. We moved on and wrapped up a few more items included in the training agenda with a good understanding of the capabilities of Live Management in Puppet Enterprise.

Overall, I found the course sufficiently challenging  and even though  I am Not a Developer I survived the full three days and can still honestly say I stand by with my assessment on the first day of class:

“The quality of the content, the level of detail provided for the exercises and the software provided are all knit together perfectly. It is immediately evident that a significant amount of effort has be invested to ensure that there are minimal hiccups and learners are not tasked with working around any technical hurdles that are not part of the primary learning outcomes. If you struggle with some part of an exercise, it is highly likely that that is exactly where they want you to experience the mental gymnastics, not an artifact of an unexplored issue.

Whether you are getting started with Puppet or a user who has not yet been provided an opportunity to attend Puppet Labs training I highly recommend the Puppet Fundamentals for System Administrators course.  It is a very good investment of your time, resources and brain power.


Day-2 of pre #PuppetConf Training: Puppet Fundamentals for System Administrators #Puppetize @PuppetConf

This is a summary of my experience on the second day of PuppetConf pre-conference training, Puppet Fundamentals for System Administrators. Day 1 was covered [ here ].

Confession time: My head hurts!

The knowledge really started flowing on this second day of training. I have to admit, I’m feeling fairly confident in my general file management and system navigation within the puppet training environment… and that’s good because there’s not time to be stumbling and fumbling over paths and parameters. We’re drinking from the firehose of nitty gritty puppetization goodness.

A Firehose called PFS:

In puppet parlance PFS stands for Package, File, and Service. These are the three primary resource types that we based a progressively more complex example exercise on. We started out simply installing apache using puppet. Then we defined where the apache config file should be generated from, then we got tricky and made the apache service check to see if the package was already installed and if it had a config file. This is all actually really simple with puppet. I just used more words to articulate what we did than it would take to actually do it using the tool. Which is not to say it’s all easy to grasp conceptually. There are a number of concepts and considerations to have in mind as you bang out your pp manifests and tests. By simple I mean the complexity of what goes on ‘under the covers’ is masked by the simplicity of puppets declarative nature and the mostly intuitive syntax.

The meat and potatoes of the morning lessons revolved around defining resources and declaring them together with dependencies, resource metaparameters such as tags, and establishing relationships between them using require, before, subscribe or notify all the while using git to push the changes up to the puppet master server so the agent could pull them down and execute them. The morning flew by and I was feeling like I had a good grasp of the various concepts coming at me and how they were applied in the exercises.

In yesterday’s post [ here ] I discussed the breakfast provided and the lack of coffee in the training rooms. This morning I decided I’d remedy both for myself. I got up a little early and walked 5 blocks down the steep hills surrounding our hotel (we’re atop Nob Hill) to a local greasy spoon diner for some bacon-laden pancakes, sausage, eggs and hash browns. My belly filled, I hiked back up those same hills toward a coffee shop and procured a ‘Traveler’ containing the life sustenance of most geeks… coffee with all the fixins! I figured after hiking those hills I really didn’t  want to be traipsing up and down multiple floors all day to get coffee and would rather share with my classmates anyhow. I think most people were grateful and thankfully nobody sounded the hotel alarm bell for sneaking in contraband. Not sure I’ll do this again tomorrow, but it’s likely.

When we broke for lunch we were informed that they would not be locking rooms as they had on day 1, which now meant leaving laptop out or carting them along to lunch. Not really a problem, but you never know…

After lunch, we started out with scoping, which I had already read about and had a general concept of top scope:, node scope: and precedence. This topic was explained very well and we had a chance to practice it again using our apache sample exercise. Also covered was variable interpolation, as in “I’m learning ${var1}${var2} in this class \n”.

Then we dove into another topic I’d read about in the getting started guide, in-out selectors and case or switch statements as well as conditionals, else/if/ifelse ,  and, or, not (these are false: undef, ‘ ‘, false. anything else will be true) Operators and regex support were thrown in, though we didn’t actually use them in an exercise. I’ll say that had I not done the pre-reading that I was able to, these topics would very likely have been beyond my reach. Definitely a good investment to grab a copy of the [ Learning Puppet docs ]

We continued to build upon our apache example to rework the web services install to support multi-platform deployment incorporating hands on application of the various topics covered.

Up until this point in the course I was feeling fairly confident that I had a handle on the concepts and didn’t stumble too hard in my attempts to put them to good use.

Then we began the last topic of the day, the concept of separating logic from presentation and dynamically generating configuration files customized for the Agent system. Read that again… it does make sense and it even sounds like a good idea… The implementation however is not exactly as drop dead simple as what I’ve seen in the rest of Puppet thus far.

The balance of the afternoon session I felt as though I was just barely teetering along the razor edge of my comfort zone. Remember, I’m not a developer… ] so wielding these dynamic, parameterized variable laden constructs is a stretch for me. That said, I’m forcing myself to reach just enough without being completely jaw-droppingly bewildered and confused.

This is what I got out of it: Puppet uses ERB templates which leverage Ruby’s built-in templating to define resource types to which you can then use to pass parameters to your modules… sounds powerful because it is, also, complex… yeah actually it is that too!

I’m still digesting a good portion of what we covered today. Maybe some sustenance and a nice cold drink will help it to all sink in.

Tomorrow is the third and final day of training. It’s also when most of the full conference attendees will be arriving, so I’m looking forward to meeting up face-to-face with several connections I’ve made via twitter in the last few weeks. If you’re here or you’re coming and you made it this far through my ramblings, hit me up @kylemurley I’m always glad to make new connections. #puppetize all the things!

Day-1 of pre #PuppetConf Training: Puppet Fundamentals for System Administrators #Puppetize @PuppetConf

This is a summary of my experience on the first day of PuppetConf pre-conference training, Puppet Fundamentals for System Administrators.

Automation Awesome: Powered by Caffeine

Since my already late night flight was delayed, I didn’t get to the hotel until well after midnight, too late to eat anything that wouldn’t have kept me up the remaining 4 hours before my wake up call. When I did wake up I was famished and considered dipping out to find some type of breakfast but thought since meals are provided during the day I’d give it a go. On the way down to the location of pre-conf training check-in I happened to get on the elevator with the event planner for Puppet. I was nicely welcomed and told it’d be fine to go on in early to the area for dining. The spread there was not bad, but definitely light; chopped fruit, danishes, muffins, juice and plenty of coffee. I’ll be going out to breakfast tomorrow though.

Speaking of coffee, there was no coffee in training rooms or sodas for that matter, only water which does a body good, but lacks the essential element that power’s much of the awesome that IT makes happen: Caffeine! Since I had met the event planner in the morning, I mentioned during a break the lack of coffee in each training room, noting that even on their own PuppetLabs Training Prerequisites site ( ) they make a point of saying there should be good coffee available. Apparently the facilities charge for coffee in each training room (there are 10+) were prohibitive, so one central location was set up. For me this was unfortunately three floors away from the rooms where my training was taking place.

Who dat?

This being my first experience really meeting the Puppet community I am making and effort to seek out contacts and find out what brings them here and what they use Puppet for in their environments. At breakfast I met a fellow attendee who works for a large corporation in the south that sells stuff on TV… all kinds of stuff… (hint: they have three letters in their name…). We chatted about what brought him to the training and how much previous experience he’d had with Puppet. Turns out his company recently acquired a hosted infrastructure shop that was already running Puppet, so he was here to learn how the heck what they’d told him could be true. They said they didn’t really have an ‘operations team’ doing the work, only a sparse staff and Puppet. That was enough of a motivator to put him on a plane to SF for a week of learning.

Also at breakfast I bumped into our trainer, Brett Gray. Who is a friendly Aussie, who once in class introduced himself as a Professional Services Engineer who previously worked in R&D at PuppetLabs and before that he ran Puppet in production environments at a customer sites.

What it be?

Let me say this very clearly, this training is really well thought out! At the hotel we’re in there are multiple training sessions going all at the same time. In our particular room are about 15 people, including Brett and Carthik another Puppet employee who is here to assist as we go along. The quality of the content, the level of detail provided for the exercises and the software provided are all knit together perfectly. It is immediately evident that a significant amount of effort has be invested to ensure that there are minimal hickups and learners are not tasked with working around any technical hurdles that are not part of the primary learning outcomes. If you struggle with some part of an exercise, it is highly likely that that is exactly where they want you to experience the mental gymnastics, not an artifact of an unexplored issue.

This stated, I’ll refer back to my previous post  [I am Not a Developer] and say there are some minimal domain knowledge that you do want to have a grasp of.  Prior to arrival onsite an email was sent to attendees pointing to a post on the Puppet Labs Training site outlining the essential skills necessary to fully benefit from the training.

As the course pre-req’s indicated, it’s essential that attendee have some type of VM virtualization platform. I found the delivery mechanism to be ideal. PuppetLabs Training even goes to the effort of providing the customized VM in multiple formats compatible with nearly every major player, VMware Workstation, Player, Fusion, OpenVM, etc.. Standing up the VM, binding it to the appropriate network and opening access to it via a bridged connection were not explicitly covered as part of the course. Out of the dozen plus students in my session there was one student in particular who was really struggling at first. The staff was incredibly patient and helpful getting this person up and running without derailing the rest of the training. I have to admit I felt a little bad for them and at the same time a bit relieved that at least it wasn’t me!

The email also advised attendees to download the training VM which was version 2.8.1. When we got to class, of course some revs had happened and the release to web was already out of date. Fortunately version 3.0 of the training VM was provided to each of us on our very own Puppet USB, much better than all trying to download it over the hotel wifi or access a share from our personal laptops.

So, what did we do?

This 3-day course covers basic system config in master-agent mode. The day began with a brief intro to the company, their recent history and the product, including the differences between Puppet Enterprise and the open source version. We’re primarily working on the CLI but the GUI is used for some exercises such as signed certs and searching for package versions across all managed nodes.

That covered, we dove into orchestration, resource abstraction layer, the great thing about idempotency and many other reasons why Puppet is the future. Lectures and slides move along at a manageable pace with short exercises to get an immediate chance at applying the material. Next thing I knew it was lunch time. The food at lunch was much better than breakfast. I met several more attendees from places as far away as Korea and Norway. We discuss how the make up of attendees really runs the gamut from large HPC federal contractors, to electronic publishers for the education industry, to large and medium enterprise web hosting all the way down to small startups who must automate to pull off what they’re doing with a skeleton crew.

Feed the Geeks

After lunch we marched ahead with the labs at full steam. We had each built out our agents, connected them to the Puppet master server and gotten our GitHub syncing our changes between the two. We then explored creating our own classes, modules and defining additional configuration parameters as well as interacting locally with Puppet apply and Puppet –Test … which doesn’t just Test, it runs the config, so watch that one!.  Once we had built our own classes it was time to learn how do we use them. The distinction between define and declare were one of the important lessons here.  To specify the contents and behavior of a class doesn’t automatically include it in a configuration; it simply makes it available to be declared. To direct Puppet to include or instantiate a given class it must be declared. To add classes you use the include keyword or the class {“foo”:} syntax.

Well, speaking of food it’s now dinner time and I’m looking forward to meeting up with some other #PuppetConf training attendees for good eats, discussion and probably an adult beverage or two. Won’t be out too late though, I want to get back into the room and try spinning up my own local instance of an all in one install of Puppet! Look for more updates tomorrow after Day Two of Puppet Fundamentals for System Administrators.

I’m learning Puppet and I am not a developer… but I sit next to one and together we automate our infrastructure

This is a bit of back story on why I am sitting here in San Francisco at my first day of PuppetConf pre-conference training, Fundamentals for System Administrators.

You see, I took enough programming as an undergraduate to gain a healthy respect for developers and to know that I am not one. Although I took some C++,  Java and a tiny bit of machine language, at some point I realized it was not necessarily my calling. In my career as an IT Pro I gravitated toward the SysAdmin and Operations side of the house. I was content to run the servers, hardware infrastructure and interact with the off the shelf software or custom apps without needing to go much further than looping through the occasional bat script, if-else and for-each that I kept in my script kitty bag of tricks. I got along well with that level of skills for nearly 15 years working for various customer organizations. Then in late 2011 I was presented with an opportunity to join a tech startup that writes software for the hypervisor storage stack. Part of the reason I decided to make the leap from being a customer to working for a partner organization was precisely because of the team of strong developers I now work alongside. Our team is lean, agile, fluid… whatever moniker you want to assign to how the work gets done, the point is things move at a quick pace and there are constantly multiple projects moving in parallel. A portion of my duties involve rapidly provisioning, configuring and managing multiple environments for everything from TestDev, QA, UAT, customer and partner POCs and of course production. With this many moving pieces, reacting to a request is already too late. I have to be proactive. Wading into this role, it quickly became evident that the ‘traditional SysAdmin’ way of working; click, click, next, next, finish was not going to be enough to keep up with my responsibilities in this environment

So there I was, painfully aware of the fact that I need to be able to make my hosts, vms, apps, infrastructure and the like come together faster and more reliably…. The question was, where to start? Realizing the potential learning opportunity I was presented with I began to take stock of what I needed to bone up on and what resources I had available to me. I needed to know how best to invest my time and energy. I knew I was not going to somehow become a developer overnight, or even in a few months. Besides, it would take me years to get where my teammates already were. They are valuable resources, especially when it comes to the coding of our infrastructure automation. I bring the operations know how of which levers to pull and buttons to push, they help me with how to coordinate the quick systematization of the otherwise labor intensive execution of repetitive tasks.

I’ll admit that working along side senior programmers can be a bit intimidating. Still, I think it’s important to realize that this relationship should actually be very complimentary. While certainly many developers are incredibly skilled, there are a great number of tasks that an infrastructure managing ‘IT guy’ does in operations that a developer may not have much experience doing. It’s important for either party not to be afraid to admit to a colleague when you don’t (yet) possess a particular skill or need their help to step through an area where they only have surface level understanding.

For myself, some of the areas I’ve been able to strengthen my skills include improving my comfort level using the commandline. Coming from a mostly Windows background, I will admit to being a bit apprehensive at first, because again, I’m not a developer… That has changed somewhat and while previously I was mostly content to remain comfortably perched atop my GUI, armed with my mouse and only a few good ole keyboard shortcuts, I now feel comfortable spending a solid portion of my time ‘shelled in’ to multiple systems at a time, monitoring, deploying and modifying systems physical and virtual, from the command line. Now, I’ll say I’m still no bash hash slash wack bang splat spittin’ CLI Wizard… but I feel I can comfortably and efficiently navigate around a file system, interact with apps, utilities and system services enough to get my job done. Piping or passing along parameters are not an insurmountable  challenge. Where I do lean on my developer is with structuring my code to be robust and with regular expressions. I had the ‘hello world’ down pat and I’d seen some awk & sed used in scripts that I might have copied and pasted somewhere without really understanding how they worked. So, PowerShell has been a great transitional tool for me and the assistance of my more wizardly colleagues, I was able to take a few scripts and cobble them together into functions.

Now, taking this another step further, we have this build system, that pulls code from sourcecontrol, does the compiling, runs some continuous integration tests and spits out a bundle with some binaries ready to be picked up and deployed…. and hey, that’s exactly why my little scripts were doing. So, we took my scripts, added some error handling, parameterized them and told the buildbot about them so it could call them every time a build was completed. This allowed for push button deployment of our app from code into an existing environment. What we’ve created can look up the latest build, check to see if it’s already running in and environment and if it isn’t, go ahead and provision it along with initiate some elementary interactions between the environment and our software. Now this is still not completely fleshed out and there are a number of enhancements that I’d like to make and have captured on Trello cards.

That is the reason I am here at PuppetConf pre-conference training. To dive headlong into learning and using Puppet to perform more advanced operations to achieve desired configuration state using the simple yet powerful declarative management tool.

So, my first takeaway for you, whether you’re a SysAdmin, operations type, or a hardcore code warrior, don’t be afraid to reach out to your  brethren. We can accomplish a lot more when we work together and pull from each other’s strengths.

Keep an eye on this space as I’ll be posting updates on my take as I progress through this 3 day adventure in Puppet.

Here’s to Day -1  of Puppet Fundamentals for System Administrators.

esxtop This is why I love the #vExpert community! Thx @Mandivs @esxtopGuru @VMwareCares @VMwFlings

When I first began working with VMware virtualization tools it was not just the technology that got me excited.

What actually gave me the most incredible feeling was the community that surrounds this awesome tech. There are innumerable examples of how time and time again I found myself either wondering about something or actually at a roadblock and the community came through for me.

A few weeks ago I had yet another of these experiences, so I thought I should get it out there as a shining example of exactly what I’m talking about.

I was working on something for my day job that required me to use the tool ‘esxtop’ – an awesome and powerful commandline utility for performance monitoring and diagnostics.  (See DuncanYB‘s posts on esxtop for a good jumping off point )

What I encountered was difficulty using the tool in the way the documentation indicated it should work.

It seemed I either didn’t understand what I was supposed to do (totally possible), or esxtop didn’t work as it claimed to.

In esxtop when monitoring storage, you have the option to expand or roll up device(s) so as to get a more detailed view of information related to the specific device(s) you’re looking at.

Once you change to the esxtop ‘u’ view (storage devices), option ‘e’ prompts you to enter the device ID.

Screenshot expand naa.jpg

It seemed that for devices with an naa.  or mpx. ID it worked fine, but with the  t10.ATA IDs it seemed like either they were being truncated somewhere or possibly had characters that were not being escaped correctly.

Screenshot naa expanded.jpg

Feeling a little frustrated (or stupid) I posted this rather cryptic tweet w/the #vExpert hash tag:

Within hours just I received a response from @VMwareCares with a viable workaround.

Armed with this nugget of knowledge I went about what I needed to do, now able to expand the t10.ATA device(s) I needed to look at.

Screenshot 2013-08-07 at 11.25.38 PM

It didn’t end there though, the next morning  Krishna Raj Raja ‏@esxtopGuru elaborated further, stating:

Raj also indicated that connecting to resxtop via vMA would be another workaround.

The help kept on coming as Manish Patel @Mandivs indicated there was a VMware KB that covered this.

So, not only by sending out a quick tweet had I found a real solution to my problem, I also learned something very useful in other areas of esxtop.  All thanks to the power of the VMware community on Twitter and thanks to @VMwareCares, Raj, and Manish.

As if that hadn’t been enough, to “top it all off, that same afternoon a new Fling was announced:

This visual tool makes esxtop even more accessible and I’ve confirmed the expand/rollup issue is definitely not a problem here.

Have you had an exceptional experience with the virtualization community over Twitter? Drop a note in the comments or point us to your write up on it. I think it’s worth recognizing that none of us go at it alone, and we do depend on each other, so a thank you is definitely in order.

Til next time, see you on the Twitters — @kylemurley