Splunk Production Services Engineer

Job Description:

At Bank of America, we are guided by a common purpose to help make financial lives better through the power of every connection. We do this by driving Responsible Growth and delivering for our clients, teammates, communities and shareholders every day.

Being a Great Place to Work and providing a culture of caring is core to how we drive Responsible Growth. We are intentional about fostering an inclusive workplace where every teammate has the opportunity to succeed, build a career and contribute to our shared success. This includes attracting and developing exceptional talent, recognizing and rewarding performance, and supporting our teammates’ physical, emotional, and financial wellness through affordable, competitive and flexible benefits.

We value the unique perspectives individuals bring from all backgrounds and career paths - whether shaped by military service, community college education, or a wide range of work and life experiences. These journeys foster resilience, leadership and innovation, strengthening our workforce and positively impact the communities we serve.

Bank of America is committed to an in-office culture that supports collaboration, engagement, and career development. Our approach includes clear in-office expectations, while providing an appropriate level of flexibility based on role-specific responsibilities and business needs.

At Bank of America, you can build a successful career with opportunities to learn, grow, and make an impact. Join us!

Position Summary:

We are seeking a highly skilled Splunk Production Services Engineer to support and operate a large-scale, business‑critical Splunk Enterprise and Splunk Cloud platform within a financial services environment.

Splunk is a foundational capability for the Information Security organization, enabling real-time security monitoring, threat detection, investigations, and regulatory reporting. This role is accountable for production stability, performance, data integrity, and security log readiness, requiring deep technical expertise and a strong operational ownership mindset.

The engineer will act as a trusted platform owner, ensuring Splunk availability, scalability, and reliability while partnering closely with Information Security, SOC, architecture, engineering, and operations teams.

Key Responsibilities:

Splunk Platform Operations & Production Stability

• Own end-to-end production support for a highly distributed Splunk Enterprise and Splunk Cloud environment, including search head clusters, indexer clusters, deployers, deployment servers, and forwarders
• Ensure high availability, performance, and resiliency of the Splunk platform supporting security and operational use cases
• Lead incident response, troubleshooting, root cause analysis (RCA), and service restoration for Splunk and Cribl platforms
• Proactively identify risks, capacity constraints, and performance bottlenecks; implement preventive and tuning measures

Security Log Ingestion & SIEM Enablement

• Serve as a key technical enabler for Information Security and SOC teams, ensuring timely, accurate, and reliable ingestion of security logs
• Onboard and normalize new data sources, supporting CIM compliance, field normalization, and SIEM best practices
• Tune ingestion pipelines using props.conf and transforms.conf, index-time and search-time optimizations
• Build and support dashboards, searches, and alerts that enable threat detection, investigations, and reporting

Cribl & Data Pipeline Management

• Administer and support the Cribl environment for data routing, filtering, enrichment, and cost optimization
• Ensure data integrity, reliability, and performance across Splunk ingestion pipelines
• Collaborate with architecture teams on data flow strategies and onboarding standards

Governance, Documentation & Compliance

• Develop and maintain runbooks, SOPs, installation guides, and operational documentation
• Adhere to change management, incident management, and SLA commitments using ITSM tools
• Operate effectively in a regulated banking environment, supporting auditability and compliance requirements

Required Qualifications:

• 5+ years of hands-on experience administering large-scale Splunk Enterprise or Splunk Cloud environments

Strong expertise in:

• Indexer clustering and search head clustering
• Universal and heavy forwarder architectures
• SmartStore / S3-compatible object storage
• SPL, search optimization, summary indexing, data model acceleration
• Deep experience with security log ingestion and SIEM use cases
• Proven ability to lead production incidents, perform RCA, and drive preventive solutions
• Strong Linux administration skills and experience managing Splunk configuration and apps
• Experience working in 24x7 production environments with high availability expectations
• Excellent written and verbal communication skills, with the ability to engage senior technical and business stakeholders

Success in this position requires:

• A production owner’s mindset
• Deep technical credibility in Splunk and data pipelines
• Ability to operate calmly and decisively during high‑severity security and platform incidents
• Strong partnership with Information Security, where Splunk availability and data quality are mission‑critical to protecting the bank

Desired Qualifications:

• Splunk certifications such as Enterprise Admin or Enterprise Architect
• Experience with Splunk Enterprise Security (ES) and SOAR (Phantom or equivalent)
• Exposure to cloud logging and security architectures (AWS, Azure, GCP)
• Knowledge of Red Hat Enterprise Linux and Windows Server administration
• Experience with monitoring, APM, and event management tools
• Strong understanding of security, network, system, and database operations
• Ability to balance multiple priorities in a fast-paced, enterprise production environment

Skills:

• Collaboration
• Influence
• Production Support
• Risk Management
• Solution Design
• Analytical Thinking
• Architecture
• Innovative Thinking
• Result Orientation
• Stakeholder Management
• Adaptability
• Automation
• DevOps Practices
• Project Management
• Solution Delivery Process

Shift:

1st shift (United States of America)

Hours Per Week: 

40

Pay Transparency details

US - NJ - Pennington - 1300 American Blvd - Hopewell Bldg 3 (NJ2130), US - VA - Richmond - 8011 VILLA PARK DR - VILLA PARK OPS CENTER BLDG B (VA2125)Pay and benefits informationPay range$92,100.00 - $160,100.00 annualized salary, offers to be determined based on experience, education and skill set.Discretionary incentive eligibleThis role is eligible to participate in the annual discretionary plan. Employees are eligible for an annual discretionary award based on their overall individual performance results and behaviors, the performance and contributions of their line of business and/or group; and the overall success of the Company.BenefitsThis role is currently benefits eligible. We provide industry-leading benefits, access to paid time off, resources and support to our employees so they can make a genuine impact and contribute to the sustainable growth of our business and the communities we serve.