Connector Development

Challenge

A monolithic ingestion system made adding new connectors difficult, requiring extensive testing for even small modifications. Customers and vendors needed a standardised framework, and file-based ingestion was introducing significant delays in data import and query performance.

Solution

Built multiple connectors using Alation's Open Connector Framework (OCF) — a gRPC-based architecture enabling modular, independent connector development. Implemented real-time gRPC streaming, incremental data ingestion, asynchronous processing, and authentication and security enhancements aligned with enterprise governance policies.

Impact

Connector deployment time reduced from months to weeks. New connectors can be added without impacting the core platform. Data ingestion speed improved 3×, leading to faster query performance and significantly reduced system latency.

Challenge

Customers were reluctant to provide admin-level access to SCM tools for security reasons. Security-conscious organisations required an on-premise solution to keep sensitive data within their VPC. Existing tools lacked observability, making it difficult to troubleshoot failures and optimise data collection.

Solution

Developed a Golang-based Connector Framework enabling deployment inside customer environments. Implemented a lightweight agent architecture with stateful incremental data ingestion, automated credential rotation, request retries, enhanced observability, and self-healing diagnostic capabilities.

Impact

Security concerns eliminated with on-premise deployments, reducing resistance from large enterprises. Incremental extraction model improved data integrity and reduced redundant network traffic. Built-in fault tolerance and auto-recovery increased reliability across all deployments.

Challenge

A scalable, automated data collection solution was needed for DORA metric calculations. The existing webhook and Kafka-based ingestion system was unreliable, often missing events and requiring high maintenance across multiple tools and topics.

Solution

Developed a REST API-based data ingestion framework supporting incremental and full data loads from multiple sources. Integrated Databricks workflows for data transformation, implemented intelligent rate-limit handling and pagination, and secured credentials via MongoDB and Vault.

Impact

Webhook failures eliminated, ensuring reliable data collection for DORA metric calculations. Intelligent request handling prevented rate-limit violations. The API-based approach reduced maintenance overhead significantly compared to the event-driven system it replaced.

Challenge

Typeface required seamless integrations with marketing, storage, CMS, and CDP platforms for automated content distribution. Salesforce Agentforce needed an AI-driven solution to enable automated, real-time content generation at scale for customer interactions.

Solution

Integrated Workato to develop real-time, API-driven workflows connecting Typeface to multiple enterprise tools. Developed a Salesforce Agentforce integration triggering Apex Classes to call Typeface AI's content generation APIs. Designed bulk data processing pipelines to reduce API load and improve efficiency.

Impact

Fully automated content publishing and marketing campaign creation. API usage optimised, reducing manual effort and infrastructure costs. Real-time content personalisation improved customer engagement and marketing performance.

Challenge

E6 required a PostgreSQL-compatible interface that would allow existing BI tools, SQL clients, and JDBC/ODBC applications to connect without native integration. The gateway needed to support PostgreSQL wire protocol semantics, metadata discovery, authentication, and query execution while seamlessly translating requests to E6.

Solution

Built a high-performance PostgreSQL wire protocol gateway with full support for PostgreSQL startup, authentication, simple and extended query flows, prepared statements, query cancellation, and connection pooling. Implemented intelligent metadata compatibility layers and protocol stubs to ensure seamless interoperability with leading BI and SQL tools, while forwarding supported queries to E6.

Impact

Enabled plug-and-play PostgreSQL connectivity for enterprise BI platforms and database clients without requiring custom drivers or application changes. Improved onboarding experience, ensured protocol-level compatibility, and provided scalable, low-latency access to E6-backed data for analytics workloads.

Challenge

As organizations adopted an expanding ecosystem of AI, analytics, and enterprise data platforms, seamless metadata integration became increasingly important. Collibra needed to rapidly grow its connector ecosystem so customers could unify metadata across their modern data stack while maintaining enterprise-grade quality, reliability, and security.

Solution

Designed and developed enterprise-grade connectors using Collibra's Edge framework, owning the complete connector lifecycle from platform evaluation and API discovery to architecture, implementation, automated validation, and production release. Enhanced the automated testing framework and streamlined the connector delivery process, enabling faster, more reliable releases across diverse enterprise platforms.

Impact

Delivered 15+ production-ready connectors across AI, business intelligence, analytics, data engineering, data integration, data warehousing, process intelligence, and enterprise data platforms, enabling organizations to seamlessly ingest metadata into Collibra. Established a scalable connector development approach that improved quality, accelerated release cycles from quarterly to biweekly, and enabled faster customer adoption.