fix: merging dev to main

[Priyanka-Microsoft]

Purpose

• ...
  This pull request implements a robust and production-ready integration of Azure Monitor Application Insights and OpenTelemetry for both infrastructure and Python backend code. The changes improve support for private networking, enable selective and safe telemetry instrumentation, and provide a reliable mechanism for tracking custom events in Application Insights. The most important changes are grouped below.

Infrastructure enhancements for Azure Monitor private networking:

• Added modules and configuration for Azure Monitor Private Link Scope (AMPLS), Data Collection Endpoint (DCE), and required private DNS zones to support secure, private ingestion and querying of monitoring data. This includes new modules, DNS zone variables, and private endpoint setup in infra/main.bicep. [1] [2] [3] [4]
• Updated data collection rules for Windows VMs to use the new DCE and removed problematic or redundant event log sources. [1] [2] [3]

Python backend telemetry and logging improvements:

• Replaced legacy Application Insights event tracking with a custom TelemetryClient-based solution for sending custom events to the correct Application Insights tables, ensuring compatibility and reliability.
• Refactored and simplified Application Insights/OpenTelemetry initialization: removed problematic or redundant configuration code, and implemented a manual, selective OpenTelemetry setup in app.py that safely instruments FastAPI HTTP requests and Python logging without interfering with async operations or OpenAI calls. [1] [2] [3] [4] [5] [6] [7]

Telemetry utility module:

• Added a new common.telemetry package with helpers for tracing and span management, providing a clean interface for future telemetry-related code.

Does this introduce a breaking change?

• Yes
• No

Golden Path Validation

• I have tested the primary workflows (the "golden path") to ensure they function correctly without errors.

Deployment Validation

• I have validated the deployment process successfully and all services are running as expected with this change.

What to Check

Verify that the following are valid

• ...

Other Information