When a bank’s AI model goes rogue, who pulls the plug, and how fast? The Reserve Bank of India’s proposed model risk framework wants an answer: continuous monitoring, verifiable audit trails, human oversight and the ability to deactivate a model the moment risk emerges. For banks, NBFCs and insurers, that moves AI governance from the compliance file to the runtime architecture.
Ganesh Narasimhadevara, Director Solutions Consulting at New Relic, tells CISO Forum that observability is the smoke detector, not the sprinkler. In this interview, he explains why one-time validation is not enough, how a five-step incident lifecycle keeps humans in charge of high-stakes decisions, and why the biggest blind spot sits in unmonitored backend agents rather than customer-facing chatbots. He also makes the case that the RBI’s engineering-first approach could become a blueprint for regulators worldwide.
The message for BFSI leaders: “good enough” monitoring may no longer be good enough.
CISO Forum: RBI’s proposed framework calls for the ability to deactivate an AI model when risk emerges. But does observability tooling actually give organisations that kill-switch control, or does the mandate require more?
Ganesh Narasimhadevara: Observability tooling provides the early warning signal rather than the physical shut-off valve. Think of observability as a smoke detector. While RBI’s mandate demands both the detector and the sprinkler system, observability serves as the detector. Intelligent observability platforms spot anomalies rapidly, whether an AI model hallucinates, generates toxic content, or breaches risk thresholds.
To meet the RBI’s mandate, organisations must integrate their AI observability platforms directly into operational layers such as API gateways, feature flags, or model orchestrators. When the observability layer flags a critical threshold breach, it fires an automated webhook to the gateway. This diverts live production traffic away from the rogue model and falls back onto deterministic, rule-based systems where a human intervenes. You cannot execute an automated kill switch without real-time observability feeding it trustworthy operational signals.
CISO Forum: What does “continuous monitoring” of an AI model look like in practice, compared with the one-time validation most organisations currently do before a model goes live, and how should that shift be understood here?
Ganesh Narasimhadevara: Traditional monitoring tools are not sufficient to monitor AI agents because one-time validation doesn’t cut it when their actions are continuous. For example, just because a person has a driver’s license doesn’t mean they can drive effectively in Bengaluru traffic. This requires driving skills and continuous assessment of which vehicle is going where and who’s violating traffic norms, and, in some cases, even gauging what drivers ahead are likely to do. Similarly, AI observability must be monitored continuously.
Unlike traditional deterministic software, AI models behave probabilistically. A model might return completely different answers to the same prompt at two different points in time. Traditional monitoring relies on point-in-time validation, which expires the moment a model encounters production inputs. In practice, continuous monitoring captures comprehensive telemetry for every single user interaction. Teams track real-time token consumption, latency spikes, semantic data drift and output quality. For a financial institution, this visibility ensures you spot immediately when an AI credit-scoring assistant suddenly begins rejecting loan applications based on biased parameters, or when a fraud detection agent flags genuine customers as mule accounts. Traditional monitoring cannot expose such agent behaviour.
CISO Forum: How does automated remediation work alongside human sign-off to ensure speed doesn’t come at the cost of accountability—and how does that fit into continuous monitoring in practice?
Ganesh Narasimhadevara: The RBI’s framework specifically calls for a human in the loop. This means organisations must enforce machine-speed diagnosis, requiring human approval for critical business processes and higher-stakes user journeys.
When performance degrades, an AI Site Reliability Engineer agent rapidly analyses system telemetry to pinpoint the root cause, slashing MTTD (mean time to detect) from hours to seconds. The system immediately restricts the blast radius by routing traffic toward pre-configured safe fallbacks. The SRE avoids unthinkingly executing code modifications directly into production. Not checking AI activity continuously drives up production incidents. Instead, intelligent observability presents the diagnostic breakdown and recommends a remediation guide to the engineer. The human acts as the gatekeeper, easily verifying and approving the fix. This preserves fast resolution times and ensures that a human makes the final decision.
CISO Forum: What would you tell a BFSI CIO who believes their existing monitoring tools are “good enough” to meet this mandate, rather than the continuous monitoring it requires, and why?
Ganesh Narasimhadevara: This is exactly why I ask CIOs to define what “good enough” means within their enterprise. Traditional application performance monitoring (APM) tools measure basic availability. AI observability evaluates behavioural sanity. A legacy APM tool tracks server uptime, CPU utilisation, memory consumption, and HTTP status codes. But for a bank’s specific customer, the support chatbot could hallucinate a policy and then promise the customer a zero per cent interest rate.
Traditional dashboards celebrate success because the server answered quickly, yet the bank faces massive systemic and regulatory fallout. Continuous AI monitoring introduces unfamiliar telemetry dimensions. Teams must track attempts at semantic prompt injection, token burn rates, semantic drift, and tool execution failures across modern agent protocols. Legacy monitoring tools stay blind to payload context, leaving enterprises exposed as they innovate.
CISO Forum: When an AI system starts behaving unexpectedly, walk us through the sequence from detection to fix—and where does a human step in along that path?
Ganesh Narasimhadevara: Enterprise AI incident management follows a five-step lifecycle:
1. Detection: The AI observability platform registers a critical anomaly, such as an internal banking assistant showing an eight-second latency spike while simultaneously leaking personally identifiable information (PII) into output telemetry.
2. Containment: Pre-configured alerting immediately triggers rules within an API gateway, instantly revoking the model’s access privileges to sensitive customer databases.
3. Investigation: The human engineer enters the loop. Rather than sifting through unstructured log volumes, the engineer reviews a distributed AI trace that highlights the specific user prompt and reveals the exact background tool that extracted confidential records.
4. Fix: The engineer analyses the data retrieval pattern, isolates the malicious edge prompt, and adjusts data-access guardrails.
5. Validation and Deployment: The team runs the update through regression and compliance testing suites before an engineer signs off and redeploys the model safely to production.
CISO Forum: How do you trace a model’s decision-making across the full AI stack, including MCP requests and tool interactions, in a way that would satisfy an auditor—and support continuous monitoring?
Ganesh Narasimhadevara: The RBI framework is explicit in calling for traceability and explainability. It rejects basic text logs showing isolated outputs. That’s because regulators understand that production AI applications actively run tools, query vector databases, and pull live records from core banking software via open standards such as the Model Context Protocol (MCP).
Meeting regulatory oversight requirements requires distributed tracing adapted specifically to AI architectures. Intelligent observability is necessary here as such platforms assign a unique trace ID at the initial user prompt. This ID propagates across the workflow, recording the retrieved context from vector databases, specific MCP API queries, intermediate model reasoning steps, and the final response. When an auditor asks why a model denied a commercial loan application, the engineering team presents a timestamped blueprint showing every piece of context the model reviewed before making its decision. That granular paper trail converts probabilistic agent decisions into fully verifiable evidence.
CISO Forum: What are the most common blind spots you see in how BFSI organisations currently monitor their AI deployments, versus what continuous monitoring requires, and which gap matters most?
Ganesh Narasimhadevara: Organisations lean heavily on aggregate dashboard metrics while completely ignoring long-tail edge failures. A team often celebrates a dashboard showing a 98% transaction success rate and sub-second average latencies. In banking, that unexamined 2% failure rate contains the regulatory sanctions, data leaks, and reputational damage. Continuous monitoring requires moving beyond high-level rollups to individual interaction telemetry to isolate root causes during minor performance shifts.
The most dangerous operational blind spot is neglecting autonomous backend agents. Financial institutions frequently over-index on monitoring public chat tools while leaving internal agents entirely unobserved. These backend worker agents integrate deeply with internal databases, orchestrate critical workflows, and access core enterprise systems. Leaving backend agents unmonitored invites substantial unmitigated risk.
CISO Forum: How do you strike the balance between machine-speed anomaly detection and ensuring humans retain meaningful oversight over high-stakes decisions as part of continuous monitoring, and what should that balance preserve?
Ganesh Narasimhadevara: Enterprises strike this balance by enforcing machine speed across operational containment while reserving policy execution for humans. Automated guardrails must function in milliseconds to halt prompt injections, filter toxic responses, or terminate runaway API loops. Humans cannot react fast enough to manually block those real-time execution hazards.
High-stakes outcomes demand human oversight. When an AI system evaluates a multi-crore corporate credit line, triggers algorithmic market positions, or denies an insurance claim, the platform must function strictly as an analytical advisor rather than an autonomous decision maker. Observability instantly feeds the human approver actionable diagnostic context and policy recommendations, reducing decision latency without forfeiting accountability. This balance preserves human responsibility for high-stakes business outcomes.
CISO Forum: What organisational or structural changes should banks, NBFCs, and insurers make internally to be ready for continuous AI monitoring at scale, and what should they prioritise first?
Ganesh Narasimhadevara: Institutions must dismantle the historical silos between compliance departments, traditional security operations, and platform engineering teams. Financial institutions previously assigned specific governance duties to separate operational units. As AI workloads spread across cross-functional enterprise workflows, cross-department ownership becomes necessary.
Enterprises should prioritise bridging the gap between regulatory intent and day-to-day engineering controls. Rather than leaving policy language in abstract compliance documents, teams must translate risk definitions directly into operational software guardrails, automated gateway webhooks, and distributed tracing architectures. Institutions experience significant hesitation over ambiguous compliance boundaries. Implementing double validation workflows and clear kill-switch triggers gives engineering teams the confidence to innovate securely.
CISO Forum: Looking at the RBI’s proposed rules as a whole, do you see this as a template other regulators globally might follow—or is India’s approach distinct in some way, given these monitoring changes?
Ganesh Narasimhadevara: India consistently demonstrates its ability to deploy robust, population-scale digital public infrastructure, as evidenced worldwide by the Unified Payments Interface (UPI). The RBI model risk management guidance represents a natural progression toward national operational resilience.
The RBI framework stands apart because it prioritises concrete engineering controls over philosophical guidelines. Many global governing bodies still produce high-level ethical manifestos and voluntary guidelines. The RBI targets runtime architecture directly, demanding continuous operational monitoring, real-time telemetry, verifiable auditability, and immediate deactivation mechanisms. By linking regulatory compliance directly to platform engineering practices, the RBI provides a practical blueprint that global central banks and financial watchdogs will closely examine as they govern enterprise AI.
