Senior Network Engineer

ESSENTIAL DUTIES AND RESPONSIBILITIES                 

Network Engineering and Administration. Manages and supports all aspects of the city’s network communications environment including, but not limited to, the following:  

• Manages and maintains the connectivity, performance, availability, and security of all city network infrastructure both on-premises and in public cloud environments (principally Microsoft Azure).  

• Delivers highly available network connectivity solutions for all city facilities and data centers as well as the internet including planning and design.  

• Responsible for the specification, procurement, service contract management, inventory management, OS/software/firmware management, lifecycle management, and replacement of all network hardware components.   

• Installs, configures, manages, and monitors all network components to ensure that they perform optimally.  

• Creates and maintains complete and accurate inventories and documentation of network systems using tables, diagrams, documents, and other suitable methods.  

• Responsible for the administration of network switching and routing, Wi-Fi controllers and access points, point-to-point wireless systems, and leased circuits.  

• Responsible for the administration of network firewalls, firewall controllers, ACLs, IPS, AAA, web content filtering, and associated regulatory compliance.  

• Responsible for the administration of remote access and site-to-site VPN solutions.  

• Responsible for network management including automation, central monitoring and notification systems, centralized logging systems, metrics and analytics, and configuration management and control systems.  

• Manages enterprise services including IP address management (IPAM), DHCP, DNS, and RADIUS.  

• Coordinates the management, operation, and monitoring of the city's data centers to ensure their high availability, reliability, and resiliency.   

• Responsible for the coordination and administration of electrical power systems supporting network hardware including the administration of uninterruptible power supply (UPS) systems, power distribution components, and collaboration with Facilities staff on provisioning suitable electrical circuits in buildings.  

• Creates and maintains emergency restoration and disaster recovery plans for the network infrastructure to assure that any component failures or damage are remediated in an orderly and timely manner.  

• Collaborates with the Chief Information Security Officer (CISO) to ensure secure network operation and compliance with organizational security guidelines and policies. Actively participates in the performance of periodic IT system security audits as well as routine, automated security vulnerability scanning, and diligently remediates detected vulnerabilities.  

• Collaborates with system administrators, application developers, and database administrators to select innovative, high performing, cost effective, efficient, scalable, and secure public cloud solutions and architectures, particularly with Azure.  

• Contributes to the migration of suitable on-premises server, storage, network, database, and application systems to IaaS, PaaS, SaaS, and serverless functions in the public cloud (Azure) and ensures their continued integration in a greater hybrid cloud architecture.  

• Collaborates with IT customers and stakeholders to understand their needs and identify opportunities to provide solutions to those needs.  

• Collaborates with the Network Architect, the architect of inside and outside physical plant infrastructure, to effectively deliver coordinated network connectivity solutions ensuring performance, high availability, scalability, and longevity.   

• Provides on-call and tier 2–3 level technical support to customers. Executes communication plans to inform customers about scheduled and unscheduled service outages and any issues affecting IT system performance, service availability, or security.  

• Creates and maintains standards and procedures for the network environment and related systems. These include best practices for equipment specification, architectures, configurations, protocols, etc.  

• Identifies requirements and designs appropriate systems solutions to meet business needs.  

• Contributes to the achievement of established service level agreements.   

• Complies with all City of Boulder user, application, and network administration security policies.  

Inside and Outside Plant Infrastructure Management. Manages and supports all aspects of the city's physical-layer network communications infrastructure including, but not limited to, the following:  

• Acts as a subject matter expert for management, planning, design, strategy, construction, maintenance, repair, contracting, documentation, and forecasting related to the city's inside physical plant (indoor copper and fiber optic cable, conduit, rack systems, and related assets) and outside physical plant (outdoor fiber optic cable, conduit, and related assets). Also serves as coordinator and collaborator on the city's partnerships, lease/trade/sale agreements, and service provider relationships as they relate to outside physical plant assets.  

• Creates and maintains complete and accurate inventories and documentation of outside plant assets and features (fiber cables, fiber strands, fiber splices, fiber terminations, conduit, manholes, handholes, poles, etc.) using tables, diagrams, fiber management systems (FMS), and geographic information systems (GIS).  

• Creates and maintains complete and accurate inventories and documentation of inside plant assets and features (copper cables, fiber cables, racks, panels, pathways, etc.) using tables, diagrams, floor plans, and other suitable methods.  

• Coordinates and manages projects for the construction, repair, and modification of physical plant infrastructure including scope definition, design collaboration, budgeting, vendor selection, contracting, purchasing, and ongoing vendor relationship management.  

• Manages a contractor for the performance of underground utility location services for all of the city's underground outside plant infrastructure. Assures that the locating contractor has continual access to accurate and up-to-date data on the location of the city's underground telecommunications assets. Supplies data to and maintains the city's relationship with the Utility Notification Center of Colorado (UNCC).  

• Creates and maintains emergency restoration and disaster recovery plans for the outside plant infrastructure to assure that any damage is remediated in an orderly and timely manner.  

• Collaborates with the Network Planner and Network Administrators to effectively deliver coordinated network connectivity solutions ensuring performance, high availability, scalability, and longevity.  

• Collaborates with other city departments including Public Works - Utilities, Public Works - Transportation, Facilities, and City Attorney for coordination of joint projects, construction permits, traffic management, facilities changes, and contracts.  

• Collaborates with partner organizations to develop and maintain mutually beneficial network connectivity solutions through agreements for shared infrastructure, shared projects, leases, trades, sales, and purchases. These partners include municipal, county, state, and federal government entities, educational institutions like the Boulder Valley School District and the University of Colorado, research institutions like NCAR, NOAA, and NIST, as well as commercial entities. Contributes to the city's lead party role in the operation of the Boulder Research and Administrative Network (BRAN) by participating in regular meetings of the BRAN partners and maintenance of the BRAN fiber network.  

• Provides on-call and tier 2–3 level technical support to customers. Executes communication plans to inform customers about scheduled and unscheduled service outages and any issues affecting IT system performance, service availability, or security.  

• Creates and maintains standards and procedures for the physical plant infrastructure and related systems. These include best practices for material specification, architectures, labeling, cable routing, etc.  

• Performs duties to achieve established service level agreements.   

Customer Service. Provides consistently excellent customer service through responsiveness, collaboration, work effort, dedication, mature and proactive communication, and conflict resolution. Customer service covers all aspects of information technology service delivery including, but not limited to, the following:  

• Quickly responding to customer incidents and working efficiently to resolve the issue while ensuring the customer has regular communication regarding the issue and a clear expectation as to next steps.   

• Providing technical assistance and information to users from other city departments on the city's technical environment, architecture, computer systems, and applications.   

• Providing technical support within the IT organization by serving as an expert-level technical resource for the Service Desk, Desktop, and Application support teams.   

• Through active communication, understanding the short- and long-term needs of the customer.  

• Analyzing historical data to recommend improvements to increase staff’s productivity.   

• Collaborating, promoting, developing, and recommending changes to the IT customer service methodology.   

Continual Improvement and Knowledge Sharing. This includes, but is not limited to, the following:   

• Serving as a leader in gaining familiarity with emerging technologies.   

• Researching new technical methods/disciplines and teaching others how to incorporate these into their day-to-day activities, work product, and style.  

• Providing guidance to the adoption of industry practices.   

• Applying knowledge and skills to collaborate, promote, develop, follow, and recommend changes to IT operations, processes, and procedures.   

• Maintaining and developing personal and professional skills and a career plan.   

• Collaborating with their supervisor to develop a career plan that meets the requirements of the division.   

• Maintaining and developing professional skills through ongoing training and career development. This is accomplished by self-study, classroom training, industry seminars, and workshops.  

Project Management. Under the functional guidance of the PMO, perform project related duties that may include, but are not limited to, the following:   

• Developing project work plans, developing project objective documents, conducting kick off meetings, monitoring and controlling project tasks and milestones, and coordinating and scheduling work on assigned projects.   

• Tracking the project and managing change control.   

• Managing work efforts related to the completion of tasks and assignments of staff assigned to the project, as well as managing work of consultants and contract employees that may be used.   

• Coordinating and tracking project budget.   

• Providing regular written and oral communications on project status via written reports, conducting regular status meetings and executive advisory board meetings with department/division heads.   

• Participating in vendor selection processes including RFP/RFI/RFQ creation and award, contract negotiations, and contract management.   

Other. This includes, but is not limited to the following:  

• Performing related duties as required to meet the needs of the city.   

• Taking proper safety precautions to prevent accidents. Responsible for the safety of self, others, materials, and equipment. Using all required safety equipment and following all safety regulations, policies, and procedures. Reporting all accidents and damage to city property.   

• Knowing and complying with all city and department policies; participating in professional trainings and development; and adhering to attendance and workplace attire policies.  

MINIMUM QUALIFICATIONS   

• Ability to demonstrate enthusiasm for furthering the role of technology in the effective delivery of important local government services.  ​

• Ability to solicit information from customers and translate business needs into technical solutions. Excellent, demonstrated written and verbal communication skills.  

• Ability to use personal vehicle (mileage will be reimbursed). Ability to participate in an on-call rotation and serve in a 24/7 on-call status 33–50% of the time. Ability to report in person to City of Boulder facilities and infrastructure sites within one hour during business hours and within four hours while on-call outside of business hours 

• Ability to manage projects incorporating formal methodologies, to train others, and to coordinate the work of team members in a team-oriented environment.  

• Knowledge and significant experience in local and wide area network planning, design, configuration, implementation, management, monitoring, troubleshooting, maintenance, and security in a large, complex, high-availability IT environment.  

• Knowledge and significant experience configuring Cisco routers, switches, firewalls, wireless LAN controllers, wireless access points, and VPN systems.  

• Knowledge of voice- and video-over-IP system components and operation.  

• Knowledge and significant experience with a variety of network management tools and protocols including SNMP. 

• Knowledge at a thorough level of and experience with IPv4 network communication, routing protocols, DNS, DHCP, RADIUS, and switched Ethernet network topologies.  

• Knowledge of and experience with network traffic management including quality of service (QoS) configuration and troubleshooting.  

• Knowledge of and significant experience with the EIGRP, OSPF, and BGP routing protocols. 

• Knowledge of and experience in data center management.  

• Knowledge of and demonstrated experience in planning, implementing, managing, and supporting inside plant structured cabling network systems.  

• Knowledge of and demonstrated experience in planning, implementing, managing, and supporting outside plant fiber optic and conduit networks including design, construction, operation, and maintenance.  

• Knowledge of and experience with the design of fiber optic cable splice plans and diagrams. High degree of proficiency in the creation of detailed diagrams with applications like Visio.  

• Skills and aptitudes for creative problem-solving, technical troubleshooting, independent learning, integrating diverse requirements, organization, attention to detail, flexibility, collaboration, customer service, management of numerous competing tasks and priorities, meeting deadlines, and working well under pressure without supervision.  

• Valid Driver’s License and ability to maintain an acceptable motor vehicle record. 

• Have and maintain acceptable background information, including criminal conviction history. 

PREFERRED QUALIFICATIONS 

• Knowledge of and experience with IP multicast. 

• Knowledge at an advanced level of and experience with Linux system administration. 

• Cisco professional- or expert-level certification or comparable network industry certification.  

• Experience with the implementation and configuration of public cloud infrastructure technologies including hybrid cloud architectures, especially with Microsoft Azure.  

• Significant experience with using scripting and programming languages for network management and automation.  

• Experience with NetFlow and related traffic analysis tools.  

• Experience with IPv6.  

• Experience with commercial or open-source network management tools for monitoring, automation, and configuration management.  

• Experience with open-source network management tools like Nagios, Icinga, Nmap, Cacti, or Zabbix. 

• Experience with security vulnerability detection including Nessus software and services.  

• Experience with using geographic information systems (GIS) for the planning and design of outside plant infrastructures in relation to other geographic features, utilities, and facilities. 

• Experience with Esri GIS software. Experience with using an FMS, especially 3-GIS, for the documentation and planning of large-scale fiber optic network infrastructures.  

• Experience and hands-on skills in the installation, testing, and active troubleshooting of inside plant and outside plant cabling networks.  

• Relevant industry certifications in outside plant or inside plant installation and testing from BICSI, FOA, other recognized certification organizations, vendors, or manufacturers.  

• Experience with managing outside plant infrastructures in a service provider environment.  

REQUIRED EDUCATION AND EXPERIENCE 

• Bachelor's degree in computer science, engineering, information technology, or a related field 

• An equivalent combination of eight (8) years’ related experience may substitute for the education requirement only 

• Five (5) or more years of recent experience as a network engineer.

Supervision Received:  Innovation and Technology Deputy Director 

Supervision Exercised:  None 

WORKING CONDITIONS AND REQUIREMENTS      

• Physical and mental effort: Sufficient clarity of speech and hearing or other communication capabilities that permits the employee to communicate effectively on the telephone and in person.  Sufficient vision or other powers of observation that permits the employee to review, evaluate and prepare a variety of written documents and materials.  Sufficient manual dexterity that permits the employee to operate computer equipment and other office equipment. Sufficient personal mobility and physical reflexes, which permits the employee to work in an office setting as well as the ability to walk on uneven surfaces and sometimes difficult terrain.   

• Work environment: Work may include extended periods of time viewing a computer video monitor and/ or operating a keyboard.  Ability to work under stress from demanding deadlines, public contact, and changing priorities and conditions.  Upper body strength to lift and carry supplies and equipment. Works primarily in a clean, comfortable environment. Works in a setting subject to continuous interruptions and background noises. 

• Machines and equipment used: Frequently uses standard office equipment including personal computers and telephones.