Mathew Fleisch

San Francisco · (415) 497-9412 · mathew.fleisch@gmail.com · resume.pdf · Blog

My name is Mathew Fleisch, and I am a Senior Infrastructure/DevOps Engineer from San Francisco. I love working on new projects and features, and often times, this means fixing and retrofitting existing products first. Every development team I have worked for, from non-profits to large corporations, have all had a finite amount of resources, whether that is in the form of time or cost. Throughout my career, I have lived by Larry Wall's philosophy and will automate as much as possible, adopting DevOps principles before there was such a title/role. I will try to automate tasks if possible, but if a task requires human input, I will create a tool that can allow someone closer to the data, to maintain it. This philosophy frees up developer time and reduces the amount of communication necessary to complete common tasks within the organization. I am also a firm believer that building stable CI/CD pipelines increases developer velocity and coupled with adequate testing coverage, can increase confidence in secure, scalable production code.
#AutomateAllTheThings

Experience

Senior Infrastructure Engineer

Workday · March 2021 - Present

The "Scylla" team at Workday maintains a platform for automating the deployment of the Workday stack on Kubernetes clusters in multi-cloud environments. This involves maintaining a number of custom built Kubernetes operators, allowing service teams to provision cloud resources (storage buckets and databases) and the automation to continuously deploy a stable product to our customers. My team has built a suite of end-to-end tests that allow us to simultaneously validate the expected functionality of each microservice and have been designed to be cloud agnostic.

Senior Infrastructure Engineer

Sysdig · Feb 2020 - 2021

I worked on developer tools in the infrastructure team at Sysdig, to increase developer velocity and productivity. Apart from participating in the on-call rotation for a multi-cloud, multi-region monitoring application, I have helped to migrate, upgrade and stabilize existing tools, and create a few of my own.

One of my first projects, was to migrate an on-prem installation of artifactory/jfrog from an ec2/docker/ebs setup to kubernetes/s3/velero. Upgrading to the latest version of artifactory, copying 1TB of artifacts to a new system, while developers continued to use the service, was challenging to coordinate; however, the outcome was successful, and unnoticed by most developers. This change was driven by cost, as the size and number of artifacts grew, the expensive aws-ebs volume had to scale as well. Using an s3 bucket for storage makes the potential artifact storage "unlimited," but required upgrading to the latest version to be a supported feature. I created scripts to copy the data from the ebs volumes to s3, verify parity, and practiced to ensure the migration would be unnoticed.

Though Sysdig is primarily a jenkins shop, I created a number of automations with GitHub Actions to plan/apply terraform, build docker containers, and entire Kops clusters, using GitOps principles. After creating "self-hosted" GitHub Action runners, I created a pipeline that would spin up a kops cluster, deploy a secondary gh-action runner, and use that secondary runner to deploy the full Sysdig stack; all triggered by merging to the main branch of a repository. These "on-demand" environments get route53 cnames to expose the web UI on a subdomain tied to a slack username, and kube-config is also made available for developers to access the cluster via kubectl. BashBot, an open-source slack bot I wrote, helps to extend the GitOps style pipeline, by triggering creation/deletion of environments, in slack. Developers type `bashbot dod-build` to create an on-demand development environment, and `bashbot dod-destroy` to tear it down.

Senior Infrastructure Engineer

Eaze · August 2019 - Jan 2020
While on the infrastructure team, we built reusable terraform to replace the legacy, snowflake dev/stage/prod environments. This allowed us to create on-demand environments for the developers to have their own personal sandboxes. These on-demand environments are decorated with tools and tests, to simulate load conditions, and have the controls/levers that operations staff would use in production. Developers are able to provision, destroy, and deploy branches, to their own environments, using the chatops bot I wrote (read more about BashBot). The bot triggers a circleci job, that builds the infrastructure via terraform in about an hour. Before I left, I was in the process of training my team to port the exiting codedeploy/ec2 portion of our pipeline, to kubernetes/helm.

Infrastructure Engineer

Eaze · November 2018 - August 2019

I had been writing tools for other engineers, while on the back-end team at Eaze, and made the transition to the infrastructure team in November of 2018. In that time I have developed many tools for the engineering department. Focusing on CI/CD pipelines, load testing, and personal development environments, as well as immersing myself into the AWS ecosystem. The technologies I have been using on this team: bash, awscli, boto3, python, javascript, chef, github-actions, circleci, buildkite, artifactory, ec2, elb/alb, s3, ecs, codedeploy, cloudwatch, cloudformation, terraform, vault, consul, helm and kubernetes.

Back-End Services Developer

Eaze · March 2018 - November 2018

Working for a start-up means constantly adapting to shifting requirements while making forward progress. In the back-end services team, I have been able to create tools to automate business tasks and help to retrofit a young application to scale efficiently.

Full Stack Developer

Apple (marketing) · May 2017 - March 2018

I was hired to work in the marketing department to maintain a sunsetting web application that was being rebuilt by another team. I made modifications to make the application more stable, secure and added logging for debugging purposes. I also expanded my role to help maintain an internal content management system.

Javascript Developer

Hitachi America · February 2017 - May 2017

I was hired for a short contract to work on an IoT analytics platform, using Javascript, NodeJS and Node Red. I wrote a user-interface in NodeJS to integrate an internal tool with Node Red to allow analytics information to be displayed about sensors and device status.

Full Stack Developer

Apple (finance) · September 2015 - October 2016

The Global Finance department at Apple created and maintained internal websites, to help facilitate secure communication and document sharing. The small team of developers primarily used LAMP Stack CodeIgniter, with some NodeJS/Grunt/Gulp/Sass optimization, for most projects, and Drupal CMS for others. Apart from other responsibilities, I wrote parallelized scripts to recover lost data for another team.

Back-End Developer

United Business Media (UBM) · November 2011 - 2015

UBM is a parent company of many conferences like Black Hat and The Game Developer's Conference. While at UBM, I developed an application to allow conference attendees to view the speaker schedule and save a personal itinerary. The Schedule Builder application is still being used by many conferences at UBM.

Staff Programmer

The Buck Institute for Research on Aging · June 2009 - 2012

Working with bioinformaticians gave me exposure to big data, automation pipelines and creating user interfaces that are easy to understand. I created web interfaces to enter large lists of genes, and various reports are displayed, based on research done at the Buck Institute.

Skills

I have personal experience with the following tools, technologies, languages and databases.

Tools / Technologies
  • AWS, GCP, Azure, IBMCloud, On-Prem
  • Terraform, CloudFormation
  • kubernetes/docker: helm, spinnaker,
    harness, argocd, fluxcd, prometheus,
    grafana, sysdig/falco
  • jenkins, github-actions, circleci,
    buildkite, codedeploy, chef, tekton,
    prow
Programming Languages
  • bash
  • python
  • golang
  • javascript/node
  • php
  • perl
  • html
  • css
Databases / Caches
  • postgres
  • mysql
  • mssql
  • nosql
  • sqlite
  • cassandra
  • elasticsearch
  • redis
  • memcache

Open-Source Examples

Projects
  • BashBot (GoLang) · source · docker-hub
    BashBot uses a json configuration file to define custom commands. Written in golang, BashBot uses slack's real-time-messaging api (RTM) to parse each message via regular expressions. If a command is detected and matched to an entry in the configuration file, bash commands are executed pertaining to that entry. Sensitive commands can be restricted to private channels, within the configuration file, and members of that private channel can execute that restricted command. This allows infrastructure teams to quickly port the internal tools they create, into version controlled slack commands, and broaden adoption to other coworkers in the organization. A slack admin can build from source, download a go-binary, or use a pre-built container via docker-hub to run BashBot in a varity of different environments. Automation builds and pushes multi-arch containers to docker hub via github actions.

  • AGIMUS (python) · source
    A community of fans of a Star Trek podcast called the Greatest Generation has a Discord server and developed a Discord bot to play games, and provide other genre specific information called AGIMUS. I helped to standardize the monolith python script, into an extensible library, that is now self-hosted in Docker or Kubernetes and now has many contributors. This application includes a mysql database (also running in a container), and has a dedicated staging discord/environment for testing new features. The repository includes github actions to build, test in pull-requests and host the container versions in github's container registry on merges to the main branch. The pull-request test validates the proposed change will not break the start-up sequence by running the bot in a KinD (Kubernetes-In-Docker) cluster on each commit to the PR. We have also developed a method to backup/restore our mysql database using a private github repository. A cron-job within the bot will run a mysql-dump four times per day, compress and push that tarball to the private repository. Other manual commands will allow us to restore from one of those commit hashes, in the case of disaster recovery. Typical functions of the bot will reach out to open-source APIs (nasa, openapi, wolfram etc) or custom made star trek themed games for discord users to interact with via slash-commands.

  • asdf version manager (bash) · source
    I have contributed to, and created a number of plugins for the asdf version manager with the goal of ensuring tools I use in self-hosted github action automations run on the raspberry pi (arm64): asdf-argocd, asdf-awscli, asdf-bashbot, asdf-cfssl, asdf-dockle, asdf-flux2, asdf-helm, asdf-helm-cr, asdf-kustomize, asdf-loki-logcli, asdf-mage, asdf-rbac-lookup, asdf-shellcheck, asdf-shfmt, asdf-tekton-cli, asdf-velero, asdf-yq


  • Build/Local Environments · github-actions-runner source · docker-dev-env source
    The docker-dev-env and github-actions-runner are similar in that they both pre-install tools that are useful for infrastructer/devops tasks, but the github-actions-runner also includes the runner agent to register a self-hosted github-action runner. I did a post about the docker-dev-env and github-actions-runner projects on my blog.

  • Timelapse Pipeline (Bash) · source
    This idea came from a hackathon project I started at Eaze with a Raspberry Pi+camera. The office looked over the ferry terminal in San Francisco, and I thought the fog and boats would be cool to see as a time lapse. Images are captured at 1fps and stored in an s3 bucket, using the date/timestamp as the filename. A processing pipeline pulls the images down in a circleci container, runs ffmpeg to make a video, and then speeds it up to match a random mp3.YouTube Channel
    Example:


  • Tetris (python) · source · mp4
    One of the unique aspects of the DEF CON conference is third-party badges that are essentially circuit boards with LEDs and screens on them. One badge has an LED matrix and an accelerometer sensor. I used the on-board interface via a serial connection to port the game Tetris to that platform in Python. This video shows Tetris in action.

  • Hak5's BashBunny Payloads (bash) · GitBunnyGit source · TwoStageMac source
    A video podcast I've subscribed to for over ten years also sells penetration testing equipment through their website. One of their products, BashBunny, combines a keyboard simulator with an arm chip running linux via USB. This allows penetration testers to script the kind of intrusion and/or exfiltration attacks, with physical access to a target computer. I have written two payloads for this device: GitBunnyGit and TwoStageMac. GitBunnyGit streamlines the process of installing and updating all other open-source payloads, by running git commands directly on the device. The TwoStageMac payload utilizes the BashBunny's ability to copy files to the target computer to run a malicious script. The sample second stage payload does some basic profiling and is intended to be swapped out for the penetration tester's own second stage.

See the Pen Column Sorting Game by Mathew Fleisch (@mathew-fleisch) on CodePen.

Programming Challenges
  • html crawler -> api -> ui (javascript) · source · demo
    This project was sparked by attending DEF CON and not liking the given interface for the conference schedule. I wrote this tool to scrape the conference website for data, and then display that data in a more effecient way.

  • Morse Code Game (arduino/c) · source
    This program uses Sparkfun's ProtoSnap board or other Arduino board with a button, buzzer, and rgb led attached to the specified pins. The concept of the program is to input morse code via a button, have a function encode, then decode a sequence of 1-5 button presses into an english character. The letter is then printed to the serial monitor and a green light blips, if a character is correctly identified; a red light blips and an error is printed to the serial monitor, if no match was found. To make game around only using the board, a user can spell "hello world" and a different tone will play, as well as a chromatic- like tone-blips as each letter is spelled correctly. If the user spells the target word (target because you the programmer, can change "hello world" to any other string of chars) incorrectly, they have to start at the beginning of the game, with the first letter.

Interests

When I am not messing with computers in some way, I like to play music, read sci-fi/fantasy, travel and eat great food.

Music: I grew up playing music and have played in performing bands most of my life. Most recently I played keyboards and saxophone in the pop punk band Our Vinyl Vows

Science Fiction/Fantasy: My favorite Authors are Robert Jordan, Brandon Sanderson, John Scalzi, Orson Scott Card, Daniel Suarez, James S.A. Corey, and Neal Stephenson. GoodReads Profile

Travel: I have visited half of the united states and went on safari in Tanzania with my wife.

Food: Whether we are in town or abroad, my family and I love to eat delicious food. We will go to restaurants or cook various dishes inspired from our families and our travels.