Scott Hazen Mueller scott@zorch.SF-Bay.ORG
61 Maple Ave
West Orange NJ 07052

Operations Tactician: Translating strategy to execution.

Professional/Managerial Skills:

Highly experienced system and network operations group manager, 1st and 2nd level. Strongly people-focused with excellent customer service skills. Significant experience in finding and solving business, management and technical problems. Highly skilled and comfortable in either team or solo situations. Forward thinker with experience in heading off and preventing problems. Effective and stable when emergencies do arise. Exceptional writing abilities and strong verbal communications skills. Extremely effective at explaining technical issues to people ranging from junior administrators to members of the Board, and at explaining business issues to technical team members. Wide technical breadth and a fast learner. Experienced at improving both efficiency and morale. Exceptional technical leader and mentor. Good at managing both to objectives and budgets. Experienced with budgeting and cost modeling at project and departmental levels. Excellent at controlling costs. Widely known and respected in the Internet industry.

Technical Skills:

20 years of problem-solving experience with multiple versions of the UNIX Operating System, Solaris 2.3-2.10, MacOS X (10.3 and 10.4), HP/UX 11.11, FreeBSD 2.1-6.1 and GNU/Linux (Redhat 6/7, Mandrake 8, Debian Sarge and Etch, Ubuntu Dapper Drake and Hardy Heron). Configuration, installation and operation of Sendmail, Popper, IMAPD, dovecot, BIND, Apache, Apache Tomcat, DHCPD, Samba, SSH, X11, XNTPD, Rsync and Netsaint/Nagios. Experience with PC hardware through Pentium-class machines with 8GB RAM and SATA or SCSI drives. Background in workstation, minicomputer and minisupercomputer systems, including SCSI, fiberchannel, SAN, NAS, DLT and other data center-grade equipment. Deep knowledge of the IP, TCP, UDP, ICMP, SMTP, POP2, POP3, IMAP, HTTP, NNTP, IRC, AIM and Telnet protocols, and operational understanding of the FTP, SNMP, NTP, DNS, DHCP, RPC, NFS, SMB, SSH and Kerberos protocols. Familiar with routing protocols. Familiar with network hardware - hubs, switches, routers, including Cisco switches and routers. Fundamentals of telecom WANs, including working knowledge of T1, T3, Frame Relay, DSL and ATM. Deep knowledge of UNIX and network security and firewalls. Programming skills in C, Bourne and Bash shells, Perl and Python. Experience with code review processes and in writing readable code. Programming experience with Web/CGI, Oracle/SQL and MySQL environments. Experience writing HTML including exposure to PHP and Javascript.


Professional Experience:

Site Reliability Engineer/Systems Administrator (May 2007 to present)
Google Inc., New York, NY:

SRE-SA for production infrastructure with a 5 9s reliability requirement. Multiple lead roles in Google's corporate support organization. Work in a sustaining operations role as lead of global team doing oncall 24x7x365 support for server infrastructure. Assess problems and respond within 5 and 30 minute SLAs. Deal with end users and other operations groups within Google. Handle any server issue as the first point of contact. Look for ongoing trends and escalate to service owners for resolution/repair. Lead weekly tactics meetings to find larger issues and respond appropriately. As lead for internal system administration support, run a metrics-gathering experiment and restructure process from a centralized group to distributed teams. Write reporting code to talk to Remedy to gather metrics. Train new systems administrators in internal systems and processes. Recruit for and run 3-month training program for technical staff designed to develop proficiency and increase promotability. As a service owner, build automated issuance system for SSL certificates, used by internal secure infrastructure to validate https/TLS and generic SSL connections. Automate process and integrate with ticketing and source code control systems so that entire process is reduced to a single command instead of multiple manual steps. Fix bugs and provide support for this system and corporate internal web redirector service. As a service team member, support internal web hosting infrastructure, including carrying pager, providing 24x7 first responder support, end-user support, toolsmithing for automated Apache configuration reloader and team expert support for Apache Tomcat.

Bruin Online Operations Supervisor (Programmer/Analyst IV) (October 2006 to May 2007)
University of California, Los Angeles, Los Angeles/Westwood, CA:

Manage team of 4 UNIX Systems Administrators responsible for operation of "Bruin Online," UCLA's in-house POP/IMAP/Webmail system supporting 35,000 mailboxes. Oversee general UNIX operations. Hire, train and retain staff. Migrate infrastructure from Sun/Solaris to Dell/Redhat Linux. Interface with engineering and user support organizations. Cooperate with and support peer department that operates and supports a Microsoft Exchange environment based on shared SAN and backup infrastructure.

Lead UNIX Systems Administrator (Operating Systems Analyst II) (March 2003 to September 2006)
California State University, Northridge, Northridge, CA:

Senior/lead systems administrator supporting University community of 30,000 students and 4,000 staff. Manage e-mail system supporting extended community and over 100,000 mailboxes. Lead 3 systems administrators managing UNIX-based WWW, e-mail, database and application servers. Manage software and hardware infrastructure projects through to completion. Negotiate, manage and track deliverables. In conjunction with middle and upper management, set team and project goals and manage performance to those goals. Participate in departmental committees setting policies in the areas of security and change control. Mentor team members in technical intricacies of UNIX systems. Interface with other groups (network, application development, user support) within IT department.

Director, Engineering & Operations (November 2002 to January 2003)
Clever Systems/Jettis.com, Westlake Village, CA:

Responsible for day-to-day operations of Linux (Redhat 7.3) servers supporting credit-card transaction processing systems and high-volume (15-20 million hits per day) web hosting for a premiere industry customer. Update and implement Netsaint and Nagios monitoring and alerting systems. Maintain system and software rollout processes. Perform a major system re-architecture and upgrade in conjunction with the corporate software development organization. Ensure prompt resolution of customer issues. Mentor junior staff.

Managing Director, Internet Engineering (August 2001 to October 2002)
Randolphs.com, LLC, CA:

Consultant on various projects, including: design and implement an ASP service deployment for a sales contact book system. Re-architect Linux servers and network. Install, configure and debug MySQL database server replication. Create extensive documentation, including network diagram, facility map, and associated text description of systems, networks and services. Analyze customer business plans and provide feedback. Debug internal mail system problems.

Director, Network Services (March 2001 to July 2001)
Netility Corporation, A-Link Network Services Division, Sunnyvale, CA:

Managed technical operations team providing ISP services to 400+ business DSL, T1 and dialup customers. Oversaw performance of in-house MIS duties. Reduced trouble-ticket backlog (100 days+) 78%. Installed and improved management reporting capabilities of ticketing system. Wrote parts-tracking system in Perl/SQL with interface into ticketing and customer database systems. Built network in new corporate HQ. Reduced staff 20% while maintaining and even improving service quality.

Vice President, Internet Technology (July 1998 to November 2000)
Asia Online Limited, Hong Kong:

Managed Hong Kong-based ISP operations - 25 people in System/Network, Technical Support and MIS departments. Increased backbone capacity by 25%, dial-up capacity by 22%. Increased Support call pickup rate from 88% to 96%. Developed a regional network design that would cut backbone costs by 25%. Used Technical Support's customer contacts to pursue cross-selling opportunities. Worked to develop a bandwidth arbitrage plan that would increase profitability without increasing backbone costs. Performed forward planning and budgeting for local and regional technical operations. Conducted cost studies of network infrastructure and technology. Engaged in fact-finding and troubleshooting visits to branch offices. Conducted due diligence on potential acquisitions. Managed technology standards, documentation and intranet processes.

Vice President, Engineering (January 1997 to June 1998)
Whole Earth Networks, San Francisco CA:

Performed project, personnel and operations management for Internet Service Provider's Engineering organization. Assisted in creation and execution of corporate strategy. Managed contractor and vendor relationships. Evaluated product and service offerings. Made hiring and firing decisions. Mentored staff. Provided technical leadership. Architected and deployed upgrades to physical plant. Designed infrastructure for new service offerings. Improved services substantially. Implemented anti-spam measures. Improved connectivity. Added third Internet Exchange to backbone. Expanded dial-up capacity by 15% and T1 port capacity by 150%. Rebuilt dial-up network. Consolidated dial-up POPs to achieve $10,000/month in cost savings.

Member of Technical Staff (November 1996 to March 1997)
Vixie Enterprises, La Honda, CA:

Manager, Internet Services and Operations (October 1995 to November 1996)
Worldview Systems Corporation, San Francisco, CA:

UNIX System and Network Administrator (January 1990 to October 1995)
Tandem Computers Inc., Cupertino, CA:

Technical Support Engineer (June 1989 to January 1990)
Ardent Computer, Sunnyvale, CA

System Support Engineer (December 1987 to June 1989)
Pyramid Technology Corporation, Mt. View, CA

Contract Consultant/CAD Systems Operator (November 1984 to December 1987)
City of Turlock Water Quality Control Facility, Turlock, CA

Education and Certifications:

B.S. cum laude in Computer Science, CSU Stanislaus, June 1986.

Publications:

Other

Selected by Network World (December 2003) as one of the "50 most powerful people in networking."

Director Emeritus, CAUCE North America; Director, CAUCE International; past President of CAUCE North America; Chairman and founder of CAUCE (www.cauce.org), the leading anti-spam advocacy organizations.

Administer personal UNIX systems since August 1987. Manage infrastructure consisting of four servers collocated in three Internet POPs, two Mac OS X servers providing support services, one RAID-5 NAS and one UNIX-based desktop workstation, running over 1000BaseTX switched network. Install and maintain mail, news, WWW and other software.

Moderate Usenet newsgroups ba.announce, comp.newprod, comp.sys.sun.announce, misc.test.moderated, news.admin.technical and rec.arts.sf.announce.

Publish "Fusion Digest" mailing list continuously since April, 1989. Run 500-user closed-membership anti-spam mailing list since March, 1995.

Contributor to the NetNews Moderator's Handbook.

Maintain and operate automatic posting 'droid' program for ba.weather.

Media appearances in: