Principal Systems Engineer - Police - City of Philadelphia

A best-in-class city that attracts best-in-class talent, Philadelphia is an incredible place to build a career. From our thriving arts scene and rich history to our culture of passion and grit, there are countless reasons to love living and working here. With a workforce of over 30,000 people, and more than 1,000 different job categories, the City of Philadelphia offers boundless opportunities to make an impact. 
As an employer, the City of Philadelphia values inclusion, integrity, innovation, empowerment, and hard work above all else. We offer a vibrant work environment, comprehensive health care and benefits, and the experience you need to grow and excel. If you’re interested in working with a passionate team of people who care about the future of Philadelphia, start here.

What We Offer:
•    Impact - The work you do here matters to millions. 
•    Growth - Philadelphia is growing, why not grow with it? 
•    Diversity & Inclusion - Find a career in a place where everyone belongs.
•    Benefits - We care about your well-being.
 

The Office of Innovation & Technology (OIT) is the central IT agency for the City of Philadelphia headed by the Chief Information Officer (CIO). OIT oversees all major information and communications technology initiatives for the City of Philadelphia - increasing the effectiveness of the information technology infrastructure, where the services provided are advanced, optimized, and responsive to the needs of the City of Philadelphia’s businesses, residents, and visitors. OIT responsibilities include: identifying the most effective approach for implementing new information technology directions throughout city government; improving the value of the city’s technology assets and the return on the city’s technology investments; ensuring data security continuity; planning for continuing operations in the event of disruption of information technology or communications services; and supporting accountable, efficient and effective government across every city department, board, commission and agency.

The PPD Platform Engineering team is responsible for the cross-platform technologies that provide technical delivery of applications and services used in day-to-day operations. The Principal Systems Engineer position is a lead technical position in the team which oversees management, administration, and operations for the department’s IT systems infrastructure and data centers. This position requires high proficiency in areas of infrastructure operations including server operating systems, Windows Active Directory, on-prem and cloud-based infrastructure virtualization, middleware and application deployment, networks, and data recovery. The individual will have an operational focus on Business Continuity, Security Risk Management, IT Auditing Compliance, IAM (Identity and Access Management, and Disaster/Data Recovery. Under the supervision of the IT Architect, the candidate will lead and mentor a team of engineers in performing analysis, design, planning, implementations, maintenance, problem-solving and enhancements on Police systems and networks. These include a combination of servers, mainframes, cloud tenants, storage arrays, LANs, WANs, VPNs, and the physical and logical components that integrate these systems together as an enterprise backbone or platform. This position includes participation in project teams at both local and central enterprise levels so subject matter expertise, professional demeanor, customer service skills, along with the ability to work within a team is necessary.  This requires creativity and resourcefulness to accomplish goals and objectives as well as applying concepts, plans, and strategies which may deviate from traditional methods and practices. The position is also expected to have the flexibility to assist in other areas as needed.  Due to the 24/7/365 operational requirements with Public Safety and Law Enforcement, this position will also require the individual to work some non-standard work hours including nights, weekends and holidays when needed. In addition, the team runs an on-call rotation

Primary Responsibilities:

• Lead efforts in architecting, configuring, validating, and implementing solutions for IT systems infrastructure.
• Participate in strategic planning efforts or platform projects by providing feedback with subject matter expertise.  
• Train and coach team members and junior technical staff with effective mentorship strategies.
• Identify and improve IAM (identify and access management) and security auditing strategies for Police systems and networks.
• Develop and improve practices for business continuity, including service resilience, data protection, backups, and disaster recovery. 
• Compile, develop, and maintain comprehensive knowledge library including operational procedures and systems documentation.
• Manage ITSM escalations for infrastructure services and operations including root cause analysis and resolution assessments.
• Support systems and data compliance per internal, state, and federal IT/Security compliance policies, guidelines, and standards.
• Perform security risk assessments, code/firmware maintenance, and mitigations for known and discovered system exploits and vulnerabilities. 
• Monitor availability, health, and performance of IT platforms and associated components.
• Oversee all aspects of hardware/product lifecycle, including provisioning, decommissioning, and asset management.
• Act as primary technical liaison between the Police Department, peer units in the City of Philadelphia, external public safety partners, and vendors.
• Other duties as needed.

Expected Competencies, Knowledge, Skills, and Abilities

• Advanced expertise in Identity and Access Management platforms and mechanisms including AzureAD/EntraID. Microsoft Active Directory Domain Services, SSO, MFA, 2FA, or similar.
• In-Depth knowledge of VMware virtualization components including vSphere, vSAN, NSX, Site Recovery Manager, and Aria Operations.
• In-Depth knowledge of endpoint protection, detection, and response platforms.
• Advanced competency in PowerShell, Bash, Batch scripting and systems automation methodologies.
• Proven understanding of VLANs, VRFs, IPAMs, switching, routing, and other network components.
• Advanced expertise Windows and Linux server operating systems including related tools and utilities.
• Expert technical knowledge in server and storage hardware including disk arrays, processors, storage mediums, and hyperconverged systems.
• Sizing, selection, and provisioning of hardware for virtualization environments.
• Familiarity with backup and DR platforms, techniques, and best practices for business continuity and service resiliency.
• Fluent understanding of SQL platforms such as Microsoft SQL, Postgres, and Oracle.
• Excellent problem-solving, analytical, and assessment skills.
• Strong communication, mentorship, and interpersonal skills.
• Ability to work nonstandard hours including nights, weekends, and holidays as needed or as on-call primary.
• Good organizational skills, attention to detail, and ability to work independently as well prioritize tasks.

•    High School Diploma or GED required. Bachelor’s degree (B.S.) or above from four-year college or university with coursework in Computer Science, Information Technology, Information Systems or related fields is desired. Based on assessment, a combination of coursework, industry certifications, and field experience can also be considered.
•    Minimum 7 years of technical experience with increasing responsibilities and technical aptitude in addition to leading teams, technical initiatives, and projects.
•    Experience in the areas of solution engineering & design, asset management, change management, capacity planning, monitoring and alerting, system security, systems code/firmware management, incident response, enterprise backup and recovery are required.
•    Certifications in areas of systems security, VMWare, operating systems, networks, and backup and recovery are a desired plus. 

Salary Range: $90,000 - $100,000

Starting salary to be determined based on experience and qualifications.

Important: To apply, candidates must provide a cover letter and a resume.

The successful applicant will be required to submit to a fingerprint background check per CJIS policy.

Discover the Perks of Being a City of Philadelphia Employee:
•    We offer Comprehensive health coverage for employees and their eligible dependents
•    Our wellness program offers eligibility into the discounted medical plan
•    Employees receive paid vacation, sick leave, and holidays
•    Generous retirement savings options are available
•    Pay off your student loans faster - As a qualifying employer, City of Philadelphia employees are eligible to participate in the Public Service Loan Forgiveness program. Join the ranks of hundreds of employees who have already benefited from this program and achieved student loan forgiveness.
•    Enjoy a Free Commute on SEPTA - Starting September 1, 2023, eligible City employees will no longer have to worry about paying for SEPTA public transportation. Whether you're a full-time, part-time, or provisional employee, you can seize the opportunity to sign up for the SEPTA Key Advantage Program and receive free Key cards for free rides on SEPTA buses, trains, trolleys, and regional rails.
•    Unlock Tuition Discounts and Scholarships - The City of Philadelphia has forged partnerships with over a dozen esteemed colleges and universities in the area, ensuring that our employees have access to a wide range of tuition discounts and scholarships. Experience savings of 10% to 40% on your educational expenses, extending not only to City employees but in some cases, spouse and dependents too!
Join the City of Philadelphia team today and seize these incredible benefits designed to enhance your financial well-being and personal growth!

*The successful candidate must be a city of Philadelphia resident within six months of hire

Effective May 22, 2023, vaccinations are no longer required for new employees that work in non-medical, non-emergency or patient facing positions with the City of Philadelphia. As a result, only employees in positions providing services that are patient-facing medical care (ex: Nurses, doctors, emergency medical personnel), must be fully vaccinated.

The City of Philadelphia is an Equal Opportunity employer and does not permit discrimination based on race, ethnicity, color, sex, sexual orientation, gender identity, religion, national origin, ancestry, age, disability, marital status, source of income, familial status, genetic information or domestic or sexual violence victim status. If you believe you were discriminated against, call the Philadelphia Commission on Human Relations at 215-686-4670 or send an email to faqpchr@phila.gov.