WTF is operations? #serverless

I just got back from the very first ever @serverlessconf in NYC.  I have a soft spot for well-curated single-track conferences, and the organizers did an incredible job.  Major kudos to @iamstan and team for pulling together such a high-caliber mix of attendees as well as presenters.

I’m really honored that they asked me to speak.  And I had a lot of fun delivering my talk!  But in all honesty, I turned it down a few times — and then agreed, and then backed out, and then agreed again at the very last moment.  I just had this feeling like the attendees weren’t going to want to hear what I was gonna say, or like we weren’t gonna be speaking the same language.

Rainbow_dash_12_by_xpesifeindx-d5giyirWhich … turned out to be mmmmostly untrue.  To the organizers’ credit, when I expressed this concern to them, they vigorously argued that they wanted me to talk *because* they wanted a heavy dose of real talk in the mix along with all the airy fairy tales of magic and success.

 

So #serverless is the new cloud or whatever

Hi, I’m grouchy and I work with operations and data and backend stuff.  I spent 3.5 years helping Parse grow from a handful of apps to over a million.  Literally building serverless before it was cool TYVM.

So when I see kids saying “the future is serverless!” and “#NoOps!” I’m like okay, that’s cute.  I’ve lived the other side of this fairytale.  I’ve seen what happens when application developers think they don’t have to care about the skills associated with operations engineering.  When they forget that no matter how pretty the abstractions are, you’re still dealing with dusty old concepts like “persistent state” and “queries” and “unavailability” and so forth, or when they literally just think they can throw money at a service to make it go faster because that’s totally how services work.

I’m going to split this up into two posts.  I’ll write up a recap of my talk in a sec, but first let’s get some things straight.  Like words.  Like operations.

What is operations?

Let’s talk about what “operations” actually means, in the year 2016, assuming a reasonably high-functioning engineering environment.

Screen Shot 2016-05-30 at 4.29.09 PM

At a macro level, operational excellence is not a role, it’s an emergent property.  It is how you get shit done.

Operations is the sum of all of the skills, knowledge and values that your company has built up around the practice of shipping and maintaining quality systems and software.  It’s your implicit values as well as your explicit values, habits, tribal knowledge, reward systems.  Everybody from tech support to product people to CEO participates in your operational outcomes, even though some roles are obviously more specialized than others.

Saying you have an ops team who is solely responsible for reliability is about as silly as saying that “HR defines and owns our company culture!”  No.  Managers and HR professionals may have particular skills and responsibilities, but culture is an emergent property and everyone contributes (and it only takes a couple bad actors to spoil the bushel).

Thinking about operational quality in terms of “a thing some other team is responsible for” is just generally not associated with great outcomes.  It leads to software engineers who are less proficient or connected to their outcomes, ops teams who get burned out, and an overall lower quality of software and services that get shipped to customers.

These are the specialized skill sets that I associate with really good operations engineers.  Do these look optional to you?

Screen Shot 2016-05-30 at 4.41.21 PM

It depends on your mission, but usually these are not particularly optional.  If you have customers, you need to care about these things.  Whether you have a dedicated ops team or not.  And you need to care about the tax it imposes on your humans too, especially when it comes to the cognitive overhead of complex systems.

So this is my definition of operations.  It doesn’t have to be your definition.  But I think it is a valuable framework for helping us reason about shipping quality software and healthy teams.  Especially given the often invisible nature of operations labor when it’s done really well.  It’s so much easier to notice and reward shipping shiny features than “something didn’t break”.

The inglorious past

Don’t get me wrong — I understand why “operations” has fallen out of favor in a lot of crowds.  I get why Google came up with “SRE” to draw a line between what they needed and what the average “sysadmin” was doing 10+ years ago.

Ops culture has a number of well-known and well-documented pathologies: hero/martyr complexes, risk aversion, burnout, etc.  I understand why this is offputting and we need to fix it.

Also, historically speaking, ops has attracted a greater proportion of nontraditional oddballs who just love debugging and building things — fewer Stanford CS PhDs, more tinkerers and liberal arts majors and college dropouts (hi).  And so they got paid less, and had less engineering discipline, and burned themselves out doing too much ad hoc labor.Rainbow_Dash_3.png

But — this is no longer our overwhelming reality, and it is certainly not the reality we are hurtling towards.  Thanks to the SRE movement, and the parallel and even more powerful & diverse open source DevOps movement, operations engineers are … engineers.  Who specialize in infrastructure.  And there’s more value than ever in empathy and fluid skill sets, in engineers who are capable of moving between disciplines and translating between specialties.  This is where the “full-stack developer” buzzword comes from.  It’s annoying, but reflects a real craving for generalist skill sets.

The BOFH stereotype is dead.  Some of the most creative cultural and technical changes in the technical landscape are being driven by the teams most identified with operations and developer tooling.  The best software engineers I know are the ones who consistently value the impact and lifecycle of the code they ship, and value deployment and instrumentation and observability.  In other words, they rock at ops stuff.

The Glorious Future

And so I think it’s time to bring back “operations” as a term of pride.  As a thing that is valued, and rewarded.  As a thing that every single person in an org understands as being critical to success.  Every organization has unique operational needs, and figuring out what they are and delivering on them takes a lot of creativity and ingenuity on both the cultural and technical front.

“Operations” comes with baggage, no doubt.  But I just don’t think that distance and denial are aRainbow_Dash_in_flightn effective approach for making something better, let alone trash talking and devaluing the skill sets that you need to deliver quality services.

You don’t make operational outcomes magically better by renaming the team “DevOps” or “SRE” or anything else.  You make it better by naming it and claiming it for what it is, and helping everyone understand how their role relates to your operational objectives.

And now that I have written this blog post I can stop arguing with people who want to talk about “DevOps Engineers” and whether “#NoOps” is a thing and maybe I can even stop trolling them back about the nascent “#NoDevs” movement.  (Haha just kidding, that one is too much fun.)

Part 2: Operations in a #Serverless World

 

WTF is operations? #serverless

Nail polish: the superior paint

Here is a thing that more people need to know: nail polish is the toughest, brightest, sparkliest, most durable paint in the world.  It makes everything prettier and brighter and it lasts forever.

I have a tradition of always painting a new keyboard every time I start a new job.  Please witness exhibit A, my freshly painted Hound keyboard & num pad:

IMG_5784
microsoft sculpt ergonomic keyboard

Doesn’t that just make you happy to look at?  The idea of typing on a plain beige or black keyboard all day long is … actually one of the more depressing things I can imagine.

Now check out the one I painted when I started at Parse and used nearly every day for almost 4 years:

IMG_5707
old-school microsoft natural keyboard, RIP

Still pretty cute, right?  It’s a little dingy and a few chips here or there but is there literally any other paint on earth that would have taken this abuse for four years?  (And you can’t even tell how pretty the shimmery holographic silver keys are in this pic.)

Dude, it gets better.  My work laptop:

IMG_5264
work laptop, aluminum MBP with hound swag

And since I got my work and personal laptops around the same time, I made them fraternal twins.

IMG_4811 (1)
personal laptop, dark grey MBA

Yo, this shit can bang around in my laptop bag for *years* and never chip or wear off or fade.  Nail polish is the most badass paint in the entire world.

(The only thing I will not paint is my fingernails, I hate painted nails.)

I really really wish I had saved pictures of all the laptops and keyboards and monitors and mice and cell phones and other crap I have painted over the last decade or so.  But here are a few of the pics and pieces I still have lying around:

 

People keep asking me where I get my laptop “stickers”, or my awesome keyboards, and now you know.

Nail polish.  It is the shit.

Pro tip #1: if you want to paint bright colors on a black background, you need to lay down a very light base first.  Surprisingly, white doesn’t work well, it’s not opaque enough and ends up looking streaky.  You need something reflective, like silver aluminum foil or silver chrome — maybe this?  Dunno, I have a big stash so I haven’t really bought anything new in years and can’t recommend specific brands.

Pro Tip #2: cheap polish is A-OK.  Those 99 cent bottles are fine, just select for opacity.  Sheer colors don’t work very well, if you have to apply more than 2-3 coats it will get globby and weird as well as being tedious.

Pro Tip #3: sketch your designs out with a pencil before you paint them (easily erasable), and apply a top coat the next day to make everything even shinier and longer-lasting.

Please send me your pics.  🙂

Happy painting!!

Nail polish: the superior paint

Two weeks with Terraform

I’ve been using terraform regularly for 2-3 weeks now.  I have terraformed in rage, I have terraformed in delight.  I thought it might be helpful to share some of my notes and lessons learned.

Why Terraform?

Because I am fucking sick and tired of not having versioned infrastructure.  Jesus christ, the ways my teams have bent over backwards to fake infra versioning after the fact (nagios checks running ec2 diffs, anyone?).

Because I am starting from scratch on a green field project, so I have the luxury of experimenting without screwing over existing customers.  Because I generally respect Hashicorp and think they’re on the right path more often than not.

If you want versioned infra, you basically get to choose between 1) AWS CloudFormation and its wrappers (sparkleformation, troposphere), 2) chef-provisioner, and 3) Terraform.

The orchestration space is very green, but I think Terraform is the standout option.  (More about why later.)  There is precious little evidence that TF was developed by or for anyone with experience running production systems at scale, but it’s … definitely not as actively hostile as CloudFormation, so it’s got that going for it.

First impressions

Stage one: my terraform experiment started out great.  I read a bunch of stuff and quickly spun up a VPC with public/private subnets, NAT, routes, IAM roles etc in < 2 days.  This would be nontrivial to do in two days *without* learning a new tool, so TOTAL JOY.

Stage two: spinning up services.  This is where I started being like … “huh.  Has anyone ever actually used this thing?  For a real thing?  In production?”  Many of the patterns that seemed obvious and correct to me about how to build robust AWS services were completely absent, like any concept of a subnet tier spanning availability zones.  I did some inexcusably horrible things with variables to get the behavior I wanted.

Stage three: … modules.  Yo, all I wanted to do was refactor a perfectly good working config into modules for VPC, security groups, IAM roles/policies/users/groups/profiles, S3 buckets/configs/policies, autoscaling groups, policies, etc, and my entire fucking world just took a dump for a week.  SURE, I was a TF noob making noob mistakes, but I could not believe how hard it was to debug literally anything..

This is when I started tweeting sad things.

The best (only) way of debugging terraform was just reading really, really carefully, copy-pasting back and forth between multiple files for hours to get all the variables/outputs/interpolation correct.  Many of the error messages lack any context or line numbers to help you track down the problem.  Take this prime specimen:

Error downloading modules: module aws_vpc: Error loading .terraform
/modules/77a846c64ead69ab51558f8c5be2cc44/main.tf: Error reading 
config for aws_route_table[private]: parse error: syntax error

Any guesses?  Turned out to be a stray ‘}’ on line 105 in a different file, which HCL vim syntax highlighting thought was A-OK.  That one took me a couple hours to track down.

Or this:

aws_security_group.zookeeper_sg: cannot parse '' as int: 
strconv.ParseInt: parsing "": invalid syntax

Which *obviously* means you didn’t explicitly define some inherited port as an int, so there’s a string somewhere there lurking in your tf tree.  (*Obviously* in retrospect, I mean, after quite a long time poking haplessly about.)

Later on I developed more sophisticated patterns for debugging terraform.  Like, uhhh, bisecting my diffs by commenting out half of the lines I had just added, then gradually re-adding or re-commenting out more lines until the error went away.

Security groups are the worst for this.  SO MANY TIMES I had security group diffs run cleanly with “tf apply”, but then claim to be modifying themselves over and over.  Sometimes I would track this down to having passed in a variable for a port number or range, e.g. “cidr_blocks = [“${var.ip_range}”]”.  Hard-coding it to “cidr_blocks [“10.0.0.0/8″]” or setting the type explicitly would resolve the problem.  Or if I accidentally entered a CIDR range that AWS didn’t like, like 10.0.20.0/20 instead of 10.0.16.0/20.  The change would apply and usually it would work, it just didn’t think it had worked, or something.  TF wasn’t aware there was a problem with the run so it would just keep “successfully” reapplying the diff every time it ran.

Some advice for TF noobs

  • As @phinze told me, “modules are basically like functions — a variable is an argument, output is a return value”.  This was helpful, because that was completely unintuitive to me when I started refactoring.  It took a few days of wrestling with profoundly inscrutable error messages before modules really clicked for me.
  • Strings.  Lists.  You can only pass variables around as strings.  Split() and join() are your friends.  Oh my god I would sell so many innocent children for the ability to pass maps back and forth between modules.
  • No interpolation for resource names makes me so sad.  Basically you can either use local variable maps, or multiple lists and just … run those index counters like a boss I guess..
  • Use AWS termination protection for stateful services or anything risky once you’re in production.  Use create_before_destroy on resources like ASG launch configs.  Use “don’t destroy” where you must — but as sparingly as possible, because that basically breaks the entire TF model.
  • If you change the launch config for an ASG, like replacing the AMI for example, you might expect TF to kick off an instance recycle.  It will not.  You must manually terminate the instances to pick up the new config.
  • If you’re collaborating with a team — ok, even if you’re not — find a remote place to store the tfstate files.  Try S3 or github, or shell out for Atlas.  Local state on laptops is for losers.
  • TF_LOG=DEBUG has never once been helpful to me.  I can only assume it was written for the Hashicorp developers, not for those of us using the product.

Errors returned by AWS are completely opaque.  Like “You were not allowed to apply this update”.  Huh?  Ok well if it fails on “tf plan”, it’s probably a bad terraform config.  If it successfully plans but fails on “tf apply”, your AWS logic is probably at fault.

Terraform does not do a great job of surfacing AWS errors.

For example, here is some terraform output:

tf output: "* aws_route_table.private: InvalidNatGatewayID.NotFound
: The natGateway ID 'nat-0e5f4ea507113b423' does not exist"

Oh!~  Okay, I go to the AWS console and track down that NAT gateway object and find this:

"Elastic IP address [eipalloc-8583b7e1] is already associated"

Hey, that seems useful!  Seems like TF just timed out bringing up one of the route tables, so it tried assigning the same EIP twice.  It would be nice to surface more of this detail into the terraform output, I hate having to resort to a web console.

Last but not least: one time I changed the comment string on a security group, and “tf plan” went into an infinite dependency loop.  I had to roll back the change, run terraform destroy against all resources in a bash for loop, and create an new security group with all new instances/ASGs just to change the comment string.  You cannot change comment strings or descriptions for resources without the resources being destroyed.  This seems PROFOUNDLY weird to me.

Wrapper scripts

Lots of people seem to eventually end up wrapping terraform with a script.  Why?

  • There is no concept of a $TF_ROOT.  If you run tf from the wrong directory, it will do some seriously confusing and screwed up shit (like duping your config, but only some of it).
  • If you’re running in production, you prob do not want people to be able to accidentally “terraform destroy” the world with the wrong environment
  • You want to enforce test/staging environments, and promotion of changes to production after they are proven good
  • You want to automatically re-run “tf plan” after “tf apply” and make sure your resources have converged cleanly.
  • So you can add slack hooks, or hipchat hooks, or github hooks.
  • Ummm, have I mentioned that TF can feel somewhat undebuggable?  Several people have told me they create rake tasks or YML templates that they then generate .tf files from so they can debug those when things break.  (Erf …)

Okay, so …..

God, it feels I’ve barely gotten started but I should probably wrap it up.[*]  Like I said, I think terraform is best in class for infra orchestration.  And orchestration is a thing that I desperately want.  Orchestration and composability are the future of infrastructure.

But also terraform is green as fuck and I would not recommend it to anyone who needs a 4-nines platform.

Simply put, there is a lot of shit I don’t want terraform touching.  I want terraform doing as little as possible.  I have already put a bunch of things into terraform that I plan on taking right back out again.  Like, you should never be running a userdata.sh script after TF has bootstrapped a node.  Yuck.. That is a job for your cfg management, or possibly a job for packer or a custom firstboot script, but never your orchestration tool!  I have already stuffed a bunch of Route53 DNS into TF and I will be ripping that right back out soon.  Terraform should not be managing any kind of dynamic data.  Or service registry, or configs, or ….

Terraform is fantastic for defining the bones of your infrastructure.  Your networking, your NAT, autoscaling groups, the bits that are robust and rarely change.  Or spinning up replicas of production on every changeset via Travis-CI or Jenkins — yay!  Do that!

But I would not feel safe making TF changes to production every day.  And you should delegate any kind of reactive scaling to ASGs or containers+scheduler or whatever.  I would never want terraform to interfere with those decisions on some arbitrary future run.

Which is why it is important to note that terraform does not play nicely with others.  It wants to own the whole thing.  Monkeypatching TF onto an existing infra is kind of horrendous.  It would be nice if you could tag certain resources or products as “this is managed by some other system, thx”.

So: why terraform?

Well, it is fairly opinionated.  It’s actively developed by some really smart people.  It’s moving fast and has most of the momentum in the space.  It’s composable and interacts well with other players iff you make some good life choices.  (Packer, for example, is amazing, by far the most unixy utility of the Hashicorp library.)

Just look at the rate of bug fixes and releases for Terraform vs CloudFormation.  Set aside crossplatform compatibility etc, and just look at the energy of the respective communities.  Not even a fair fight.

Want more?  Ok, well I would rather adopt one opinionated philosophy for my infrastructure, supplementing where necessary, than duct tape together fifty different half baked philosophies about how software and infrastructure should work and spend all my time mediating their conflicts.  (This is one of my beefs with CloudFormation: AWS has no opinions, only slobbering, squidlike, directionless-flopping optionalities.  And while we’re on the topic it also has nothing like “tf plan” for previewing changes, so THAT’S PRETTY STUPID TOO.)

I do have some concerns about Hashicorp spreading themselves too thin on too many products.  Some of those products probably shouldn’t exist.  Meh.

Terraform has a ways to go before it feels predictable and debuggable, but I think it’s heading in the right direction.  It’s been a fun couple weeks and I’m excited to start contributing to the ecosystem and integrating with other components, like chef-solo & consul.

 

[*] OMGGGGGGG, I never even got to the glorious horrors of the terraforming gem and how you are most definitely going to end up manually editing your *.tfstate files.  Ahahahahaa.

[**] Major thanks to @phinze, @solarce, @ascendantlogic, @lusis, @progrium and others who helped me limp through my first few weeks.

Two weeks with Terraform

How to survive an acquisition

Last week Facebook announced that they will be shutting down Parse.

 

I have so, so many feelings.  But this isn’t about my feelings now.  This is about what I wish I had known when we got acquired.

When they told us we were getting acquired, it felt like a wrecking ball to the gut.  Nothing against Facebook, it just wasn’t what I signed up for.  Big company, bureaucracy, 3-4 hour daily commute, and goddammit Parse was my *baby*.  What the fuck was this big company going to do to my baby?

Here’s the first thing I wish I had known: this is normal.  Trauma is the norm.  Acquisitions are *always* massively disruptive and upsetting, even the good ones.

This was all compounded by the jarring disconnect between internal and external perceptions of the acquisition.  The surreality of being written up in the press and having everyone act like this was the greatest thing that ever happened to us, versus the shock and dismay and confusion that many of us were actually experiencing.

Here are a few more things I wish I had understood.

You don’t own your product anymore.  Your product is now “strategic alignment”.  Look at yourself in the mirror and repeat that five times every morning.

Your customers are not your customers anymore.  Your customer is now your corporate overlord.  You can resist this and try to serve your old customers first, but it will wear your engineering team out and eventually drive you mad.

Cultures will clash.  Yours will lose.  You *must* learn the tribal and hierarchal games of the new org if you want to succeed.  They don’t make sense to you, and you didn’t choose into them, but you must learn them anyway if you want to succeed.

Assume good intent.  Aggressively assume good intent.  If a big company buys up your tiny startup, lots of people are going to condescend to you and assume they know how to solve your own problems better than you do.  In retrospect, I realize that a lot of tiny acquired startups *don’t* know what the fuck they’re doing and so the behemoth just gets used to assuming that everyone is like that.  Just grit your teeth and take it.

And if you honestly can’t see yourself ever embracing the new parent company, whether for cultural or ethical or technical reasons, you should leave sooner rather than later.  (UNLESS it’s for the golden handcuffs in which case for god’s sake try to emotionally disengage and do not be a people manager.)

One of Facebook’s internal slogans is something like “always do what’s best for Facebook”.  I never gave a shit about Facebook’s mission; I cared about Parse’s.  I remain incredibly proud of everything we accomplished for Parse, MongoDB, and RocksDB.  If only our strategic alignments had, well, aligned.

 

How to survive an acquisition

2015 in recaps and life changes

2015 is dead, long live 2016

A pretty serious list of ridiculous/amazing shit happened in 2015.  My face was on a fucking pillar.  I got to meet KEITH BOSTIC.  A bunch of my favorite engineers got dressed up in cheetah kigurumi pajamas and ran around playing booth babes.  And that’s just a three day sampling of mischief.

IMG_4213 (1)
THIS IS MY FACE ON A PILLAR.  you are allowed to find this incredibly creepy.

There are at least three key things about 2015 that I feel like I am always going to associate with this year, so that’s what I want to write about.  (I’ll do a boring second post later with links to all the talks, articles etc for the year.)

The three things that stood out for me this year were:

  1. Conference Overload
  2. MongoDB + RocksDB (aka “mongo grows up”)
  3. Leaving Parse.

The Year of Conference Insanity

In 2015 I gave talks at 21 conferences, which was … excessive.

This wasn’t really intentional.  But I was bored and restless and unhappy a lot and had many erratically available spare cycles.  Being a manager at a big company makes this depressing Swiss cheese out of your calendar.  You can’t really carve out the heads-down time you would need in order to do challenging technical work, but it’s easy to go on lots of short trips to conferences.  And travel is nice.  So I just kind of said “yes” to everything.

I really appreciate all the brilliant people and organizers who welcomed me to their events this year, and all the many random, unexpectedly lovely connections I made there.

Next year will be very different.

IMG_4135 (1)
that one day i packed lighter fluid in to work to burn things because reasons

MongoDB + RocksDB

When I started running MongoDB in 2012, we were on version 2.0 and it had a single lock.  Yep, just the one!  This was a SUPER fun way to try and build a sophisticated multi-tenant platform.

Mongo was rolling out performance improvements with every release.  But shit actually got real when they put the storage engine API on the roadmap and acquired WiredTiger.  The Facebook RocksDB team enthusiastically pitched in too, providing feedback on the API design and ultimately delivering a full-fledged implementation of MongoDB with RocksDB storage engine.

IMG_4203
the RocksDB “marketing” team, with bonus Peter Zaitsev

Parse spent the past 1.5 years being the alpha customer for MongoRocks.  We developed a load testing framework for replaying production workloads, evaluated TokuMX and WiredTiger as well as Rocks, worked with the RocksDB team (aka Igor Canadi) to iron the kinks out of the MongoRocks implementation and figure out how to run and monitor the damn thing, and then got it rolled it out to 100% of our production replica sets in under a year.  (Worth it? with Rocks we used 1/10th the storage space and writes were 50-200x faster compared to mmap.)

IMG_4286
Asya, Domas and Mark Callaghan at the inaugural MongoDB Storage Engine Summit

I’ve said this before but I really believe it: 2015 is the year that MongoDB grew up and became a “real database”.  This was Mongo’s InnoDB moment.  Remember how much shit everyone used to give MySQL back in the MyISAM days?  Hey guess what, software actually can get better!  This is one reason it has been a special delight getting to work with Mark Callaghan.  Mark was one of those engineers who turned MySQL from a punch line into a “real database” which powers much of the internet.  So it was kind of like having this guy around.

I loved having a front row seat to mongo’s awkward adolescent phase.  I loved getting to play a small role in promoting storage engine diversity.  With Percona now offering enterprise support for both MyRocks and MongoRocks, I think the project is in good hands, and I’m really really fucking proud of this.

Leaving Parse

I said goodbye to Parse in early September.  This was one of those emotionally overwhelming life changes that somehow manages to feel both too soon and way, way overdue.  I spent 3.5 years working on Parse — one year pre-acquisition, 2.5 years at Facebook.

IMG_4631 (1)
my going-away cake.  yes, it is a unicorn shitting rainbows with a purple tutu and sparkly silver slippers.  thank you nancy!!

I like to think of myself as this crusty backend engineer who sighs wearily and doesn’t give a shit about product, but the fact is I fell in love with Parse on the very first day.  I am so, so proud of what we built.  We built a thing that people really cared about.  We empowered so many developers to build and create and make whole new businesses on our platform.

There is nothing as heady as getting to pour yourself into a product that people are passionate about, tackle problems that are hard and might actually be impossible, with a team that blows your socks off with their brilliance and joy and hilariousness on a daily basis.

IMG_3726
the amazing parse production engineering team.  my boys 4 lyfe

So yeah, it was hard to let go.  But it was time.  The 3-4 hour commute was fucking killing me, and it had been a long, *long* time since I felt like I was learning new things or pushing my boundaries as an engineer or a leader.  I need a certain amount of chaos and panic in my life.  Being comfortable for too long just makes me miserable.

Parse is going to be a hard act to follow.  Thanks for raising the bar, motherfuckers.

IMG_5114
quitting day

In summary

2015 was ridiculous and awesome, and I want 2016 to be ridiculous and awesome but in some radical and fantastically new ways.  I have many hopes and intentions for 2016.  but I’ll save that for another time.

Oh but my single greatest achievement of 2015?  is *clearly* this tattoo.

IMG_4939

12 months, 11 appts, about 30 hours of inking.  thank you micah riot.

2015 in recaps and life changes

Hello world

“Start a blog” has been on my todo list for so long, it’s going to leave a gaping void in my life when I check it off.

There are some things I have been wanting to talk about for a while now, in a way that’s less ephemeral than twitter.  Like how the tech industry is terrible at interviewing and hiring, and some ideas for making it slightly less terrible.  Or the appalling proliferation of new ways to bootstrap your infrastructure over the past couple years.  Tips for how to survive when your startup gets acquired.  Stuff like that.

The first thing on the list tho will be a wrap-up of all the talks and articles and shit that I did in 2015, because Caitie did it and it was amazing and inspiring.  ❤

This post is basically for me to try and figure out how to use wordpress themes.  But here, just so that it isn’t 100% devoid of content let me state for the record where “@mipsytipsy” comes from.  When I was in school we had an assembly language class where we had to use SPIM to emulate the MIPS R3000 processor.  The class was wretched, but “spimmy” became my irc nick.  During my Everquest years I played an enchanter named Mipsy Tipsy, and now I’m just too lazy to come up with anything new.

A surprising number of people still call me “Spimmy” in real life.  Heh.

IMG_3485

photo credit: i don’t remember whose hands these were.  maybe @mosheroperandi?

Image