fix: exception filter clear (#1936)

* feat: backend changes for supporting exception filters

* feat: frontend changes for exception page filter support

* chore: extractSingleFilterValue is updated

* fix: handle frontend edge case

Co-authored-by: Ankit Nayan <ankit@signoz.io>
Co-authored-by: Palash Gupta <palashgdev@gmail.com>

.github/workflows/e2e-k3s.yaml

@@ -16,7 +16,9 @@ jobs:
         uses: actions/checkout@v2
 
       - name: Build query-service image
-        run: make build-query-service-amd64
+        env:
+          DEV_BUILD: 1
+        run: make build-ee-query-service-amd64
 
       - name: Build frontend image
         run: make build-frontend-amd64

Makefile

@@ -120,19 +120,15 @@ down-local:
 	down -v
 
 run-x86:
-	@docker-compose -f \
-	$(STANDALONE_DIRECTORY)/docker-compose-core.yaml -f $(STANDALONE_DIRECTORY)/docker-compose-prod.yaml \
-	up --build -d
+	@docker-compose -f $(STANDALONE_DIRECTORY)/docker-compose.yaml up --build -d
 
 down-x86:
-	@docker-compose -f \
-	$(STANDALONE_DIRECTORY)/docker-compose-core.yaml -f $(STANDALONE_DIRECTORY)/docker-compose-prod.yaml \
-	down -v
+	@docker-compose -f $(STANDALONE_DIRECTORY)/docker-compose.yaml down -v
 
 clear-standalone-data:
 	@docker run --rm -v "$(PWD)/$(STANDALONE_DIRECTORY)/data:/pwd" busybox \
-	sh -c "cd /pwd && rm -rf alertmanager/* clickhouse/* signoz/*"
+	sh -c "cd /pwd && rm -rf alertmanager/* clickhous*/* signoz/* zookeeper-*/*"
 
 clear-swarm-data:
 	@docker run --rm -v "$(PWD)/$(SWARM_DIRECTORY)/data:/pwd" busybox \
-	sh -c "cd /pwd && rm -rf alertmanager/* clickhouse/* signoz/*"
+	sh -c "cd /pwd && rm -rf alertmanager/* clickhous*/* signoz/* zookeeper-*/*"

README.md

@@ -25,17 +25,25 @@
 
 SigNoz helps developers monitor applications and troubleshoot problems in their deployed applications. SigNoz uses distributed tracing to gain visibility into your software stack.
 
+👉 Visualise Metrics, Traces and Logs in a single pane of glass
+
 👉 You can see metrics like p99 latency, error rates for your services, external API calls and individual end points.
 
 👉 You can find the root cause of the problem by going to the exact traces which are causing the problem and see detailed flamegraphs of individual request traces.
 
 👉 Run aggregates on trace data to get business relevant metrics
 
-![screenzy-1644432902955](https://user-images.githubusercontent.com/504541/153270713-1b2156e6-ec03-42de-975b-3c02b8ec1836.png)
+👉 Filter and query logs, build dashboards and alerts based on attributes in logs
+
+![screenzy-1670570187181](https://user-images.githubusercontent.com/504541/206646629-829fdafe-70e2-4503-a9c4-1301b7918586.png)
 <br />
-![screenzy-1644432986784](https://user-images.githubusercontent.com/504541/153270725-0efb73b3-06ed-4207-bf13-9b7e2e17c4b8.png)
+![screenzy-1670570193901](https://user-images.githubusercontent.com/504541/206646676-a676fdeb-331c-4847-aea9-d1cabf7c47e1.png)
 <br />
-![screenzy-1647005040573](https://user-images.githubusercontent.com/504541/157875938-a3d57904-ea6d-4278-b929-bd1408d7f94c.png)
+![screenzy-1670570199026](https://user-images.githubusercontent.com/504541/206646754-28c5534f-0377-428c-9c6e-5c7c0d9dd22d.png)
+<br />
+![screenzy-1670569888865](https://user-images.githubusercontent.com/504541/206645819-1e865a56-71b4-4fde-80cc-fbdb137a4da5.png)
+
+
 
 <br /><br />
 
@@ -51,12 +59,12 @@ Come say Hi to us on [Slack](https://signoz.io/slack) 👋
 
 ## Features:
 
+- Unified UI for metrics, traces and logs. No need to switch from Prometheus to Jaeger to debug issues, or use a logs tool like Elastic separate from your metrics and traces stack. 
 - Application overview metrics like RPS, 50th/90th/99th Percentile latencies, and Error Rate
 - Slowest endpoints in your application
 - See exact request trace to figure out issues in downstream services, slow DB queries, call to 3rd party services like payment gateways, etc
 - Filter traces by service name, operation, latency, error, tags/annotations.
 - Run aggregates on trace data (events/spans) to get business relevant metrics. e.g. You can get error rate and 99th percentile latency of `customer_type: gold` or `deployment_version: v2` or `external_call: paypal`
-- Unified UI for metrics and traces. No need to switch from Prometheus to Jaeger to debug issues.
 
 <br /><br />
 
@@ -78,6 +86,12 @@ We support [OpenTelemetry](https://opentelemetry.io) as the library which you ca
 - Python
 - NodeJS
 - Go
+- PHP
+- .NET
+- Ruby
+- Elixir
+- Rust
+
 
 You can find the complete list of languages here - https://opentelemetry.io/docs/
 
@@ -123,6 +137,21 @@ Moreover, SigNoz has few more advanced features wrt Jaeger:
 - Jaegar UI doesn’t show any metrics on traces or on filtered traces
 - Jaeger can’t get aggregates on filtered traces. For example, p99 latency of requests which have tag - customer_type='premium'. This can be done easily on SigNoz
 
+<p>&nbsp  </p>
+
+### SigNoz vs Elastic 
+
+- SigNoz Logs management are based on ClickHouse, a columnar OLAP datastore which makes aggregate log analytics queries much more efficient
+- 50% lower resource requirement compared to Elastic during ingestion
+
+<p>&nbsp  </p>
+
+### SigNoz vs Loki
+
+- SigNoz supports aggregations on high-cardinality data over a huge volume while loki doesn’t.
+- SigNoz supports indexes over high cardinality data and has no limitations on the number of indexes, while Loki reaches max streams with a few indexes added to it.
+- Searching over a huge volume of data is difficult and slow in Loki compared to SigNoz
+
 <br /><br />
 
 <img align="left" src="https://signoz-public.s3.us-east-2.amazonaws.com/Contributors.svg" width="50px" />

README.zh-cn.md

@@ -13,14 +13,19 @@
 
 ##
 
-SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNoz使用分布式跟踪来增加软件技术栈的可见性。
+SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNoz使用分布式追踪来增加软件技术栈的可见性。
 
-👉 你能看到一些性能矩阵，服务、外部api调用、每个终端(endpoint)的p99延迟和错误率。
+👉 你能看到一些性能指标，服务、外部api调用、每个终端(endpoint)的p99延迟和错误率。
 
-👉 通过准确的跟踪来确定是什么引起了问题，并且可以看到每个独立请求的帧图(framegraph)，这样你就能找到根本原因。
+👉 通过准确的追踪来确定是什么引起了问题，并且可以看到每个独立请求的帧图(framegraph)，这样你就能找到根本原因。
 
+👉 聚合trace数据来获得业务相关指标。
 
-![SigNoz Feature](https://signoz-public.s3.us-east-2.amazonaws.com/signoz_hero_github.png)
+![screenzy-1644432902955](https://user-images.githubusercontent.com/504541/153270713-1b2156e6-ec03-42de-975b-3c02b8ec1836.png)
+<br />
+![screenzy-1644432986784](https://user-images.githubusercontent.com/504541/153270725-0efb73b3-06ed-4207-bf13-9b7e2e17c4b8.png)
+<br />
+![screenzy-1647005040573](https://user-images.githubusercontent.com/504541/157875938-a3d57904-ea6d-4278-b929-bd1408d7f94c.png)
 
 <br /><br />
 
@@ -36,12 +41,12 @@ SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNo
 
 ## 功能:
 
-- 应用总览矩阵(matrix)，如RPS, 50/90/99百分比延迟率，错误率
+- 应用概览指标(metrics)，如RPS, p50/p90/p99延迟率分位值，错误率等。
 - 应用中最慢的终端(endpoint)
-- 查看准确的网络请求跟踪来分析下游服务问题、慢数据库查询问题 及调用第三方服务如支付网关的问题
-- 通过服务名称、操作、延迟、错误、标签来过滤跟踪
-- 对过滤后的跟踪数据做矩阵聚合。比如，获得过滤条件`customer_type: gold` or `deployment_version: v2` or `external_call: paypal`的错误率和p99延迟
-- 整合的矩阵和跟踪用户界面。不需要像从Prometheus切换到Jaeger才能调试问题
+- 查看特定请求的trace数据来分析下游服务问题、慢数据库查询问题 及调用第三方服务如支付网关的问题
+- 通过服务名称、操作、延迟、错误、标签来过滤traces。
+- 聚合trace数据(events/spans)来得到业务相关指标。比如，你可以通过过滤条件`customer_type: gold` or `deployment_version: v2` or `external_call: paypal` 来获取指定业务的错误率和p99延迟
+- 为metrics和trace提供统一的UI。排查问题不需要在Prometheus和Jaeger之间切换。
 
 <br /><br />
 
@@ -53,7 +58,7 @@ SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNo
 
 我们想做一个自服务的开源版本的工具，类似于DataDog和NewRelic，用于那些对客户数据流入第三方有隐私和安全担忧的厂商。
 
-开源也让你对配置、采样和上线率有完整的控制，你可以在SigNoz基础上构建模块来满足特定的商业需求。
+开源也让你对配置、采样和正常运行时间有完整的控制，你可以在SigNoz基础上构建模块来满足特定的商业需求。
 
 ### 语言支持
 
@@ -71,8 +76,8 @@ SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNo
 <img align="left" src="https://signoz-public.s3.us-east-2.amazonaws.com/Philosophy.svg" width="50px" />
 
 ## 入门
-  
-  
+
+
 ### 使用Docker部署
 
 请按照[这里](https://signoz.io/docs/deployment/docker/)列出的步骤使用Docker来安装
@@ -80,35 +85,34 @@ SigNoz帮助开发人员监控应用并排查已部署应用中的问题。SigNo
 如果你遇到任何问题，这个[排查指南](https://signoz.io/docs/deployment/troubleshooting)会对你有帮助。
 
 <p>&nbsp  </p>
-  
-  
+
+
 ### 使用Helm在Kubernetes上部署
 
 请跟着[这里](https://signoz.io/docs/deployment/helm_chart)的步骤使用helm charts安装
-  
 
 <br /><br />
 
 <img align="left" src="https://signoz-public.s3.us-east-2.amazonaws.com/UseSigNoz.svg" width="50px" />
 
-## Comparisons to Familiar Tools
+## 与其他方案的比较
 
 ### SigNoz vs Prometheus
 
-如果你只是需要矩阵，那Prometheus是不错的，但如果你要无缝的在矩阵和跟踪之间切换，那目前把Prometheus & Jaeger串起来的体验并不好。
+如果你只是需要监控指标(metrics)，那Prometheus是不错的，但如果你要无缝的在metrics和traces之间切换，那目前把Prometheus & Jaeger串起来的体验并不好。
 
-我们的目标是在矩阵和跟踪之间提供整合的UI - 类似于Datadog这样的Saas厂提供的方案，能够对跟踪进行过滤和聚合，这是目前Jaeger缺失的功能。
+我们的目标是为metrics和traces提供统一的UI - 类似于Datadog这样的Saas厂提供的方案。并且能够对trace进行过滤和聚合，这是目前Jaeger缺失的功能。
 
 <p>&nbsp  </p>
 
 ### SigNoz vs Jaeger
 
-Jaeger只做分布式跟踪，SigNoz则是做了矩阵和跟踪两块，我们在计划中也有日志管理功能。
+Jaeger只做分布式追踪(distributed tracing)，SigNoz则支持metrics,traces,logs ，即可视化的三大支柱。
 
 并且SigNoz有一些Jaeger没有的高级功能：
 
-- Jaegar UI无法在跟踪或过滤的跟踪基础上展示矩阵。
-- Jaeger不能在过滤的跟踪上进行聚合操作。例如，拥有tag为customer_type='premium'的所有请求的p99延迟，在SigNoz里这很容易实现。
+- Jaegar UI无法在traces或过滤的traces上展示metrics。
+- Jaeger不能对过滤的traces做聚合操作。例如，拥有tag为customer_type='premium'的所有请求的p99延迟。而这个功能在SigNoz这儿是很容易实现。
 
 <br /><br />
 
@@ -121,6 +125,23 @@ Jaeger只做分布式跟踪，SigNoz则是做了矩阵和跟踪两块，我们
 
 还不清楚怎么开始？ 只需在[slack社区](https://signoz.io/slack)的`#contributing`频道里ping我们。
 
+### Project maintainers
+
+#### Backend
+
+- [Ankit Nayan](https://github.com/ankitnayan)
+- [Nityananda Gohain](https://github.com/nityanandagohain)
+- [Srikanth Chekuri](https://github.com/srikanthccv)
+- [Vishal Sharma](https://github.com/makeavish)
+
+#### Frontend
+
+- [Palash Gupta](https://github.com/palashgdev)
+
+#### DevOps
+
+- [Prashant Shahi](https://github.com/prashant-shahi)
+
 <br /><br />
 
 <img align="left" src="https://signoz-public.s3.us-east-2.amazonaws.com/DevelopingLocally.svg" width="50px" />

deploy/docker-swarm/clickhouse-setup/clickhouse-cluster.xml

@@ -0,0 +1,75 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
+         Optional. If you don't use replicated tables, you could omit that.
+
+         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
+      -->
+    <zookeeper>
+        <node index="1">
+            <host>zookeeper-1</host>
+            <port>2181</port>
+        </node>
+        <!-- <node index="2">
+            <host>zookeeper-2</host>
+            <port>2181</port>
+        </node>
+        <node index="3">
+            <host>zookeeper-3</host>
+            <port>2181</port>
+        </node> -->
+    </zookeeper>
+
+    <!-- Configuration of clusters that could be used in Distributed tables.
+         https://clickhouse.com/docs/en/operations/table_engines/distributed/
+      -->
+    <remote_servers>
+        <cluster>
+            <!-- Inter-server per-cluster secret for Distributed queries
+                 default: no secret (no authentication will be performed)
+
+                 If set, then Distributed queries will be validated on shards, so at least:
+                 - such cluster should exist on the shard,
+                 - such cluster should have the same secret.
+
+                 And also (and which is more important), the initial_user will
+                 be used as current user for the query.
+
+                 Right now the protocol is pretty simple and it only takes into account:
+                 - cluster name
+                 - query
+
+                 Also it will be nice if the following will be implemented:
+                 - source hostname (see interserver_http_host), but then it will depends from DNS,
+                   it can use IP address instead, but then the you need to get correct on the initiator node.
+                 - target hostname / ip address (same notes as for source hostname)
+                 - time-based security tokens
+            -->
+            <!-- <secret></secret> -->
+            <shard>
+                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
+                <!-- <internal_replication>false</internal_replication> -->
+                <!-- Optional. Shard weight when writing data. Default: 1. -->
+                <!-- <weight>1</weight> -->
+                <replica>
+                    <host>clickhouse</host>
+                    <port>9000</port>
+                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
+                    <!-- <priority>1</priority> -->
+                </replica>
+            </shard>
+            <!-- <shard>
+                <replica>
+                    <host>clickhouse-2</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+            <shard>
+                <replica>
+                    <host>clickhouse-3</host>
+                    <port>9000</port>
+                </replica>
+            </shard> -->
+        </cluster>
+    </remote_servers>
+</clickhouse>

deploy/docker-swarm/clickhouse-setup/clickhouse-config.xml

@@ -236,8 +236,8 @@
     <openSSL>
         <server> <!-- Used for https server AND secure tcp port -->
             <!-- openssl req -subj "/CN=localhost" -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /etc/clickhouse-server/server.key -out /etc/clickhouse-server/server.crt -->
-            <certificateFile>/etc/clickhouse-server/server.crt</certificateFile>
-            <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile>
+            <!-- <certificateFile>/etc/clickhouse-server/server.crt</certificateFile> -->
+            <!-- <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile> -->
             <!-- dhparams are optional. You can delete the <dhParamsFile> element.
                  To generate dhparams, use the following command:
                   openssl dhparam -out /etc/clickhouse-server/dhparam.pem 4096
@@ -618,148 +618,6 @@
     </jdbc_bridge>
     -->
 
-    <!-- Configuration of clusters that could be used in Distributed tables.
-         https://clickhouse.com/docs/en/operations/table_engines/distributed/
-      -->
-    <remote_servers>
-        <!-- Test only shard config for testing distributed storage -->
-        <test_shard_localhost>
-            <!-- Inter-server per-cluster secret for Distributed queries
-                 default: no secret (no authentication will be performed)
-
-                 If set, then Distributed queries will be validated on shards, so at least:
-                 - such cluster should exist on the shard,
-                 - such cluster should have the same secret.
-
-                 And also (and which is more important), the initial_user will
-                 be used as current user for the query.
-
-                 Right now the protocol is pretty simple and it only takes into account:
-                 - cluster name
-                 - query
-
-                 Also it will be nice if the following will be implemented:
-                 - source hostname (see interserver_http_host), but then it will depends from DNS,
-                   it can use IP address instead, but then the you need to get correct on the initiator node.
-                 - target hostname / ip address (same notes as for source hostname)
-                 - time-based security tokens
-            -->
-            <!-- <secret></secret> -->
-
-            <shard>
-                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
-                <!-- <internal_replication>false</internal_replication> -->
-                <!-- Optional. Shard weight when writing data. Default: 1. -->
-                <!-- <weight>1</weight> -->
-                <replica>
-                    <host>localhost</host>
-                    <port>9000</port>
-                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
-                    <!-- <priority>1</priority> -->
-                </replica>
-            </shard>
-        </test_shard_localhost>
-        <test_cluster_one_shard_three_replicas_localhost>
-            <shard>
-                <internal_replication>false</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.3</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <!--shard>
-                <internal_replication>false</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.3</host>
-                    <port>9000</port>
-                </replica>
-            </shard-->
-        </test_cluster_one_shard_three_replicas_localhost>
-        <test_cluster_two_shards_localhost>
-             <shard>
-                 <replica>
-                     <host>localhost</host>
-                     <port>9000</port>
-                 </replica>
-             </shard>
-             <shard>
-                 <replica>
-                     <host>localhost</host>
-                     <port>9000</port>
-                 </replica>
-             </shard>
-        </test_cluster_two_shards_localhost>
-        <test_cluster_two_shards>
-            <shard>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-        </test_cluster_two_shards>
-        <test_cluster_two_shards_internal_replication>
-            <shard>
-                <internal_replication>true</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <internal_replication>true</internal_replication>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-        </test_cluster_two_shards_internal_replication>
-        <test_shard_localhost_secure>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>9440</port>
-                    <secure>1</secure>
-                </replica>
-            </shard>
-        </test_shard_localhost_secure>
-        <test_unavailable_shard>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>1</port>
-                </replica>
-            </shard>
-        </test_unavailable_shard>
-    </remote_servers>
-
     <!-- The list of hosts allowed to use in URL-related storage engines and table functions.
         If this section is not present in configuration, all hosts are allowed.
     -->
@@ -786,29 +644,6 @@
          Values for substitutions are specified in /clickhouse/name_of_substitution elements in that file.
       -->
 
-    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
-         Optional. If you don't use replicated tables, you could omit that.
-
-         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
-      -->
-
-    <!--
-    <zookeeper>
-        <node>
-            <host>example1</host>
-            <port>2181</port>
-        </node>
-        <node>
-            <host>example2</host>
-            <port>2181</port>
-        </node>
-        <node>
-            <host>example3</host>
-            <port>2181</port>
-        </node>
-    </zookeeper>
-    -->
-
     <!-- Substitutions for parameters of replicated tables.
           Optional. If you don't use replicated tables, you could omit that.
 

deploy/docker-swarm/clickhouse-setup/docker-compose.yaml

@@ -1,30 +1,127 @@
 version: "3.9"
 
+x-clickhouse-defaults: &clickhouse-defaults
+  image: clickhouse/clickhouse-server:22.8.8-alpine
+  tty: true
+  deploy:
+    restart_policy:
+      condition: on-failure
+  depends_on:
+    - zookeeper-1
+    # - zookeeper-2
+    # - zookeeper-3
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+  healthcheck:
+    # "clickhouse", "client", "-u ${CLICKHOUSE_USER}", "--password ${CLICKHOUSE_PASSWORD}", "-q 'SELECT 1'"
+    test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"]
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+
+x-clickhouse-depend: &clickhouse-depend
+  depends_on:
+    - clickhouse
+    # - clickhouse-2
+    # - clickhouse-3
+
 services:
+  zookeeper-1:
+    image: bitnami/zookeeper:3.7.0
+    hostname: zookeeper-1
+    user: root
+    ports:
+      - "2181:2181"
+      - "2888:2888"
+      - "3888:3888"
+    volumes:
+      - ./data/zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      # - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+
+  # zookeeper-2:
+  #   image: bitnami/zookeeper:3.7.0
+  #   hostname: zookeeper-2
+  #   user: root
+  #   ports:
+  #     - "2182:2181"
+  #     - "2889:2888"
+  #     - "3889:3888"
+  #   volumes:
+  #     - ./data/zookeeper-2:/bitnami/zookeeper
+  #   environment:
+  #     - ZOO_SERVER_ID=2
+  #     - ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
+  #     - ALLOW_ANONYMOUS_LOGIN=yes
+  #     - ZOO_AUTOPURGE_INTERVAL=1
+
+  # zookeeper-3:
+  #   image: bitnami/zookeeper:3.7.0
+  #   hostname: zookeeper-3
+  #   user: root
+  #   ports:
+  #     - "2183:2181"
+  #     - "2890:2888"
+  #     - "3890:3888"
+  #   volumes:
+  #     - ./data/zookeeper-3:/bitnami/zookeeper
+  #   environment:
+  #     - ZOO_SERVER_ID=3
+  #     - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
+  #     - ALLOW_ANONYMOUS_LOGIN=yes
+  #     - ZOO_AUTOPURGE_INTERVAL=1
+
   clickhouse:
-    image: clickhouse/clickhouse-server:22.4.5-alpine
+    <<: *clickhouse-defaults
+    hostname: clickhouse
     # ports:
     #   - "9000:9000"
     #   - "8123:8123"
-    tty: true
+    #   - "9181:9181"
     volumes:
       - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
       - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
       # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
       - ./data/clickhouse/:/var/lib/clickhouse/
-    deploy:
-      restart_policy:
-        condition: on-failure
-    logging:
-      options:
-        max-size: 50m
-        max-file: "3"
-    healthcheck:
-      # "clickhouse", "client", "-u ${CLICKHOUSE_USER}", "--password ${CLICKHOUSE_PASSWORD}", "-q 'SELECT 1'"
-      test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
+
+  # clickhouse-2:
+  #   <<: *clickhouse-defaults
+  #   hostname: clickhouse-2
+  #   ports:
+  #     - "9001:9000"
+  #     - "8124:8123"
+  #     - "9182:9181"
+  #   volumes:
+  #     - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
+  #     - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+  #     - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
+  #     # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  #     - ./data/clickhouse-2/:/var/lib/clickhouse/
+
+  # clickhouse-3:
+  #   <<: *clickhouse-defaults
+  #   hostname: clickhouse-3
+  #   ports:
+  #     - "9002:9000"
+  #     - "8125:8123"
+  #     - "9183:9181"
+  #   volumes:
+  #     - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
+  #     - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+  #     - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
+  #     # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  #     - ./data/clickhouse-3/:/var/lib/clickhouse/
 
   alertmanager:
     image: signoz/alertmanager:0.23.0-0.2
@@ -40,7 +137,7 @@ services:
         condition: on-failure
 
   query-service:
-    image: signoz/query-service:0.11.1
+    image: signoz/query-service:0.12.0
     command: ["-config=/root/config/prometheus.yml"]
     # ports:
     #   - "6060:6060"     # pprof port
@@ -66,11 +163,10 @@ services:
     deploy:
       restart_policy:
         condition: on-failure
-    depends_on:
-      - clickhouse
+    <<: *clickhouse-depend
 
   frontend:
-    image: signoz/frontend:0.11.1
+    image: signoz/frontend:0.12.0
     deploy:
       restart_policy:
         condition: on-failure
@@ -83,7 +179,7 @@ services:
       - ../common/nginx-config.conf:/etc/nginx/conf.d/default.conf
 
   otel-collector:
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-config.yaml"]
     user: root # required for reading docker container logs
     volumes:
@@ -91,6 +187,7 @@ services:
       - /var/lib/docker/containers:/var/lib/docker/containers:ro
     environment:
       - OTEL_RESOURCE_ATTRIBUTES=host.name={{.Node.Hostname}},os.type={{.Node.Platform.OS}},dockerswarm.service.name={{.Service.Name}},dockerswarm.task.name={{.Task.Name}}
+      - DOCKER_MULTI_NODE_CLUSTER=false
     ports:
       # - "1777:1777"     # pprof extension
       - "4317:4317"     # OTLP gRPC receiver
@@ -107,11 +204,10 @@ services:
       mode: global
       restart_policy:
         condition: on-failure
-    depends_on:
-      - clickhouse
+    <<: *clickhouse-depend
 
   otel-collector-metrics:
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-metrics-config.yaml"]
     volumes:
       - ./otel-collector-metrics-config.yaml:/etc/otel-collector-metrics-config.yaml
@@ -123,8 +219,7 @@ services:
     deploy:
       restart_policy:
         condition: on-failure
-    depends_on:
-      - clickhouse
+    <<: *clickhouse-depend
 
   hotrod:
     image: jaegertracing/example-hotrod:1.30

deploy/docker-swarm/clickhouse-setup/otel-collector-config.yaml

@@ -47,7 +47,7 @@ receivers:
       # thrift_binary:
       #   endpoint: 0.0.0.0:6832
   hostmetrics:
-    collection_interval: 60s
+    collection_interval: 30s
     scrapers:
       cpu: {}
       load: {}
@@ -55,6 +55,16 @@ receivers:
       disk: {}
       filesystem: {}
       network: {}
+  prometheus:
+    config:
+      global:
+        scrape_interval: 60s
+      scrape_configs:
+        # otel-collector internal metrics
+        - job_name: otel-collector
+          static_configs:
+          - targets:
+            - localhost:8888
 
 processors:
   batch:
@@ -65,11 +75,10 @@ processors:
     # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
     detectors: [env, system] # include ec2 for AWS, gce for GCP and azure for Azure.
     timeout: 2s
-    override: false
   signozspanmetrics/prometheus:
     metrics_exporter: prometheus
     latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
-    dimensions_cache_size: 10000
+    dimensions_cache_size: 100000
     dimensions:
       - name: service.namespace
         default: default
@@ -94,6 +103,8 @@ processors:
 exporters:
   clickhousetraces:
     datasource: tcp://clickhouse:9000/?database=signoz_traces
+    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
+
   clickhousemetricswrite:
     endpoint: tcp://clickhouse:9000/?database=signoz_metrics
     resource_to_telemetry_conversion:
@@ -103,6 +114,7 @@ exporters:
   # logging: {}
   clickhouselogsexporter:
     dsn: tcp://clickhouse:9000/
+    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
     timeout: 5s
     sending_queue:
       queue_size: 100
@@ -134,8 +146,8 @@ service:
       receivers: [otlp]
       processors: [batch]
       exporters: [clickhousemetricswrite]
-    metrics/hostmetrics:
-      receivers: [hostmetrics]
+    metrics/generic:
+      receivers: [hostmetrics, prometheus]
       processors: [resourcedetection, batch]
       exporters: [clickhousemetricswrite]
     metrics/spanmetrics:

deploy/docker-swarm/clickhouse-setup/otel-collector-metrics-config.yaml

@@ -2,27 +2,19 @@ receivers:
   prometheus:
     config:
       scrape_configs:
-        # otel-collector internal metrics
-        - job_name: "otel-collector"
-          scrape_interval: 60s
-          dns_sd_configs:
-            - names:
-                - 'tasks.otel-collector'
-              type: 'A'
-              port: 8888
         # otel-collector-metrics internal metrics
-        - job_name: "otel-collector-metrics"
+        - job_name: otel-collector-metrics
           scrape_interval: 60s
           static_configs:
             - targets:
               - localhost:8888
         # SigNoz span metrics
-        - job_name: "signozspanmetrics-collector"
+        - job_name: signozspanmetrics-collector
           scrape_interval: 60s
           dns_sd_configs:
             - names:
-                - 'tasks.otel-collector'
-              type: 'A'
+                - tasks.otel-collector
+              type: A
               port: 8889
 
 processors:

deploy/docker-swarm/common/nginx-config.conf

@@ -30,6 +30,8 @@ server {
 
     location /api {
         proxy_pass http://query-service:8080/api;
+        # connection will be closed if no data is read for 600s between successive read operations
+        proxy_read_timeout 600s; 
     }
 
     # redirect server error pages to the static page /50x.html

deploy/docker/clickhouse-setup/clickhouse-cluster.xml

@@ -0,0 +1,75 @@
+<?xml version="1.0"?>
+<clickhouse>
+    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
+         Optional. If you don't use replicated tables, you could omit that.
+
+         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
+      -->
+    <zookeeper>
+        <node index="1">
+            <host>zookeeper-1</host>
+            <port>2181</port>
+        </node>
+        <!-- <node index="2">
+            <host>zookeeper-2</host>
+            <port>2181</port>
+        </node>
+        <node index="3">
+            <host>zookeeper-3</host>
+            <port>2181</port>
+        </node> -->
+    </zookeeper>
+
+    <!-- Configuration of clusters that could be used in Distributed tables.
+         https://clickhouse.com/docs/en/operations/table_engines/distributed/
+      -->
+    <remote_servers>
+        <cluster>
+            <!-- Inter-server per-cluster secret for Distributed queries
+                 default: no secret (no authentication will be performed)
+
+                 If set, then Distributed queries will be validated on shards, so at least:
+                 - such cluster should exist on the shard,
+                 - such cluster should have the same secret.
+
+                 And also (and which is more important), the initial_user will
+                 be used as current user for the query.
+
+                 Right now the protocol is pretty simple and it only takes into account:
+                 - cluster name
+                 - query
+
+                 Also it will be nice if the following will be implemented:
+                 - source hostname (see interserver_http_host), but then it will depends from DNS,
+                   it can use IP address instead, but then the you need to get correct on the initiator node.
+                 - target hostname / ip address (same notes as for source hostname)
+                 - time-based security tokens
+            -->
+            <!-- <secret></secret> -->
+            <shard>
+                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
+                <!-- <internal_replication>false</internal_replication> -->
+                <!-- Optional. Shard weight when writing data. Default: 1. -->
+                <!-- <weight>1</weight> -->
+                <replica>
+                    <host>clickhouse</host>
+                    <port>9000</port>
+                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
+                    <!-- <priority>1</priority> -->
+                </replica>
+            </shard>
+            <!-- <shard>
+                <replica>
+                    <host>clickhouse-2</host>
+                    <port>9000</port>
+                </replica>
+            </shard>
+            <shard>
+                <replica>
+                    <host>clickhouse-3</host>
+                    <port>9000</port>
+                </replica>
+            </shard> -->
+        </cluster>
+    </remote_servers>
+</clickhouse>

deploy/docker/clickhouse-setup/clickhouse-config.xml

@@ -236,8 +236,8 @@
     <openSSL>
         <server> <!-- Used for https server AND secure tcp port -->
             <!-- openssl req -subj "/CN=localhost" -new -newkey rsa:2048 -days 365 -nodes -x509 -keyout /etc/clickhouse-server/server.key -out /etc/clickhouse-server/server.crt -->
-            <certificateFile>/etc/clickhouse-server/server.crt</certificateFile>
-            <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile>
+            <!-- <certificateFile>/etc/clickhouse-server/server.crt</certificateFile> -->
+            <!-- <privateKeyFile>/etc/clickhouse-server/server.key</privateKeyFile> -->
             <!-- dhparams are optional. You can delete the <dhParamsFile> element.
                  To generate dhparams, use the following command:
                   openssl dhparam -out /etc/clickhouse-server/dhparam.pem 4096
@@ -618,148 +618,6 @@
     </jdbc_bridge>
     -->
 
-    <!-- Configuration of clusters that could be used in Distributed tables.
-         https://clickhouse.com/docs/en/operations/table_engines/distributed/
-      -->
-    <remote_servers>
-        <!-- Test only shard config for testing distributed storage -->
-        <test_shard_localhost>
-            <!-- Inter-server per-cluster secret for Distributed queries
-                 default: no secret (no authentication will be performed)
-
-                 If set, then Distributed queries will be validated on shards, so at least:
-                 - such cluster should exist on the shard,
-                 - such cluster should have the same secret.
-
-                 And also (and which is more important), the initial_user will
-                 be used as current user for the query.
-
-                 Right now the protocol is pretty simple and it only takes into account:
-                 - cluster name
-                 - query
-
-                 Also it will be nice if the following will be implemented:
-                 - source hostname (see interserver_http_host), but then it will depends from DNS,
-                   it can use IP address instead, but then the you need to get correct on the initiator node.
-                 - target hostname / ip address (same notes as for source hostname)
-                 - time-based security tokens
-            -->
-            <!-- <secret></secret> -->
-
-            <shard>
-                <!-- Optional. Whether to write data to just one of the replicas. Default: false (write data to all replicas). -->
-                <!-- <internal_replication>false</internal_replication> -->
-                <!-- Optional. Shard weight when writing data. Default: 1. -->
-                <!-- <weight>1</weight> -->
-                <replica>
-                    <host>localhost</host>
-                    <port>9000</port>
-                    <!-- Optional. Priority of the replica for load_balancing. Default: 1 (less value has more priority). -->
-                    <!-- <priority>1</priority> -->
-                </replica>
-            </shard>
-        </test_shard_localhost>
-        <test_cluster_one_shard_three_replicas_localhost>
-            <shard>
-                <internal_replication>false</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.3</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <!--shard>
-                <internal_replication>false</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-                <replica>
-                    <host>127.0.0.3</host>
-                    <port>9000</port>
-                </replica>
-            </shard-->
-        </test_cluster_one_shard_three_replicas_localhost>
-        <test_cluster_two_shards_localhost>
-             <shard>
-                 <replica>
-                     <host>localhost</host>
-                     <port>9000</port>
-                 </replica>
-             </shard>
-             <shard>
-                 <replica>
-                     <host>localhost</host>
-                     <port>9000</port>
-                 </replica>
-             </shard>
-        </test_cluster_two_shards_localhost>
-        <test_cluster_two_shards>
-            <shard>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-        </test_cluster_two_shards>
-        <test_cluster_two_shards_internal_replication>
-            <shard>
-                <internal_replication>true</internal_replication>
-                <replica>
-                    <host>127.0.0.1</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <internal_replication>true</internal_replication>
-                <replica>
-                    <host>127.0.0.2</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-        </test_cluster_two_shards_internal_replication>
-        <test_shard_localhost_secure>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>9440</port>
-                    <secure>1</secure>
-                </replica>
-            </shard>
-        </test_shard_localhost_secure>
-        <test_unavailable_shard>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>9000</port>
-                </replica>
-            </shard>
-            <shard>
-                <replica>
-                    <host>localhost</host>
-                    <port>1</port>
-                </replica>
-            </shard>
-        </test_unavailable_shard>
-    </remote_servers>
-
     <!-- The list of hosts allowed to use in URL-related storage engines and table functions.
         If this section is not present in configuration, all hosts are allowed.
     -->
@@ -786,29 +644,6 @@
          Values for substitutions are specified in /clickhouse/name_of_substitution elements in that file.
       -->
 
-    <!-- ZooKeeper is used to store metadata about replicas, when using Replicated tables.
-         Optional. If you don't use replicated tables, you could omit that.
-
-         See https://clickhouse.com/docs/en/engines/table-engines/mergetree-family/replication/
-      -->
-
-    <!--
-    <zookeeper>
-        <node>
-            <host>example1</host>
-            <port>2181</port>
-        </node>
-        <node>
-            <host>example2</host>
-            <port>2181</port>
-        </node>
-        <node>
-            <host>example3</host>
-            <port>2181</port>
-        </node>
-    </zookeeper>
-    -->
-
     <!-- Substitutions for parameters of replicated tables.
           Optional. If you don't use replicated tables, you could omit that.
 

deploy/docker/clickhouse-setup/docker-compose-core.yaml

@@ -2,7 +2,7 @@ version: "2.4"
 
 services:
   clickhouse:
-    image: clickhouse/clickhouse-server:22.4.5-alpine
+    image: clickhouse/clickhouse-server:22.8.8-alpine
     container_name: clickhouse
     # ports:
     # - "9000:9000"
@@ -41,7 +41,7 @@ services:
   # Notes for Maintainers/Contributors who will change Line Numbers of Frontend & Query-Section. Please Update Line Numbers in `./scripts/commentLinesForSetup.sh` & `./CONTRIBUTING.md`
   otel-collector:
     container_name: otel-collector
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-config.yaml"]
     # user: root # required for reading docker container logs
     volumes:
@@ -67,7 +67,7 @@ services:
 
   otel-collector-metrics:
     container_name: otel-collector-metrics
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-metrics-config.yaml"]
     volumes:
       - ./otel-collector-metrics-config.yaml:/etc/otel-collector-metrics-config.yaml

deploy/docker/clickhouse-setup/docker-compose-prod.yaml

@@ -1,44 +0,0 @@
-version: "2.4"
-
-services:
-  query-service:
-    image: signoz/query-service:0.11.1
-    container_name: query-service
-    command: ["-config=/root/config/prometheus.yml"]
-    # ports:
-    #   - "6060:6060"     # pprof port
-    #   - "8080:8080"     # query-service port
-    volumes:
-      - ./prometheus.yml:/root/config/prometheus.yml
-      - ../dashboards:/root/config/dashboards
-      - ./data/signoz/:/var/lib/signoz/
-    environment:
-      - ClickHouseUrl=tcp://clickhouse:9000/?database=signoz_traces
-      - ALERTMANAGER_API_PREFIX=http://alertmanager:9093/api/
-      - SIGNOZ_LOCAL_DB_PATH=/var/lib/signoz/signoz.db
-      - DASHBOARDS_PATH=/root/config/dashboards
-      - STORAGE=clickhouse
-      - GODEBUG=netdns=go
-      - TELEMETRY_ENABLED=true
-      - DEPLOYMENT_TYPE=docker-standalone-amd
-    restart: on-failure
-    healthcheck:
-      test: ["CMD", "wget", "--spider", "-q", "localhost:8080/api/v1/version"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
-    depends_on:
-      clickhouse:
-        condition: service_healthy
-
-  frontend:
-    image: signoz/frontend:0.11.1
-    container_name: frontend
-    restart: on-failure
-    depends_on:
-      - alertmanager
-      - query-service
-    ports:
-      - "3301:3301"
-    volumes:
-      - ../common/nginx-config.conf:/etc/nginx/conf.d/default.conf

deploy/docker/clickhouse-setup/docker-compose.yaml

@@ -1,28 +1,135 @@
 version: "2.4"
 
+x-clickhouse-defaults: &clickhouse-defaults
+  restart: on-failure
+  image: clickhouse/clickhouse-server:22.8.8-alpine
+  tty: true
+  depends_on:
+    - zookeeper-1
+    # - zookeeper-2
+    # - zookeeper-3
+  logging:
+    options:
+      max-size: 50m
+      max-file: "3"
+  healthcheck:
+    # "clickhouse", "client", "-u ${CLICKHOUSE_USER}", "--password ${CLICKHOUSE_PASSWORD}", "-q 'SELECT 1'"
+    test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"]
+    interval: 30s
+    timeout: 5s
+    retries: 3
+  ulimits:
+    nproc: 65535
+    nofile:
+      soft: 262144
+      hard: 262144
+
+x-clickhouse-depend: &clickhouse-depend
+  depends_on:
+    clickhouse:
+      condition: service_healthy
+    # clickhouse-2:
+    #   condition: service_healthy
+    # clickhouse-3:
+    #   condition: service_healthy
+
 services:
+  
+  zookeeper-1:
+    image: bitnami/zookeeper:3.7.0
+    container_name: zookeeper-1
+    hostname: zookeeper-1
+    user: root
+    ports:
+      - "2181:2181"
+      - "2888:2888"
+      - "3888:3888"
+    volumes:
+      - ./data/zookeeper-1:/bitnami/zookeeper
+    environment:
+      - ZOO_SERVER_ID=1
+      # - ZOO_SERVERS=0.0.0.0:2888:3888,zookeeper-2:2888:3888,zookeeper-3:2888:3888
+      - ALLOW_ANONYMOUS_LOGIN=yes
+      - ZOO_AUTOPURGE_INTERVAL=1
+
+  # zookeeper-2:
+  #   image: bitnami/zookeeper:3.7.0
+  #   container_name: zookeeper-2
+  #   hostname: zookeeper-2
+  #   user: root
+  #   ports:
+  #     - "2182:2181"
+  #     - "2889:2888"
+  #     - "3889:3888"
+  #   volumes:
+  #     - ./data/zookeeper-2:/bitnami/zookeeper
+  #   environment:
+  #     - ZOO_SERVER_ID=2
+  #     - ZOO_SERVERS=zookeeper-1:2888:3888,0.0.0.0:2888:3888,zookeeper-3:2888:3888
+  #     - ALLOW_ANONYMOUS_LOGIN=yes
+  #     - ZOO_AUTOPURGE_INTERVAL=1
+
+  # zookeeper-3:
+  #   image: bitnami/zookeeper:3.7.0
+  #   container_name: zookeeper-3
+  #   hostname: zookeeper-3
+  #   user: root
+  #   ports:
+  #     - "2183:2181"
+  #     - "2890:2888"
+  #     - "3890:3888"
+  #   volumes:
+  #     - ./data/zookeeper-3:/bitnami/zookeeper
+  #   environment:
+  #     - ZOO_SERVER_ID=3
+  #     - ZOO_SERVERS=zookeeper-1:2888:3888,zookeeper-2:2888:3888,0.0.0.0:2888:3888
+  #     - ALLOW_ANONYMOUS_LOGIN=yes
+  #     - ZOO_AUTOPURGE_INTERVAL=1
+
   clickhouse:
-    image: clickhouse/clickhouse-server:22.4.5-alpine
-    # ports:
-    # - "9000:9000"
-    # - "8123:8123"
-    tty: true
+    <<: *clickhouse-defaults
+    container_name: clickhouse
+    hostname: clickhouse
+    ports:
+      - "9000:9000"
+      - "8123:8123"
+      - "9181:9181"
     volumes:
       - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
       - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+      - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
       # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
       - ./data/clickhouse/:/var/lib/clickhouse/
-    restart: on-failure
-    logging:
-      options:
-        max-size: 50m
-        max-file: "3"
-    healthcheck:
-      # "clickhouse", "client", "-u ${CLICKHOUSE_USER}", "--password ${CLICKHOUSE_PASSWORD}", "-q 'SELECT 1'"
-      test: ["CMD", "wget", "--spider", "-q", "localhost:8123/ping"]
-      interval: 30s
-      timeout: 5s
-      retries: 3
+
+  # clickhouse-2:
+  #   <<: *clickhouse-defaults
+  #   container_name: clickhouse-2
+  #   hostname: clickhouse-2
+  #   ports:
+  #     - "9001:9000"
+  #     - "8124:8123"
+  #     - "9182:9181"
+  #   volumes:
+  #     - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
+  #     - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+  #     - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
+  #     # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  #     - ./data/clickhouse-2/:/var/lib/clickhouse/
+
+  # clickhouse-3:
+  #   <<: *clickhouse-defaults
+  #   container_name: clickhouse-3
+  #   hostname: clickhouse-3
+  #   ports:
+  #     - "9002:9000"
+  #     - "8125:8123"
+  #     - "9183:9181"
+  #   volumes:
+  #     - ./clickhouse-config.xml:/etc/clickhouse-server/config.xml
+  #     - ./clickhouse-users.xml:/etc/clickhouse-server/users.xml
+  #     - ./clickhouse-cluster.xml:/etc/clickhouse-server/config.d/cluster.xml
+  #     # - ./clickhouse-storage.xml:/etc/clickhouse-server/config.d/storage.xml
+  #     - ./data/clickhouse-3/:/var/lib/clickhouse/
 
   alertmanager:
     image: signoz/alertmanager:0.23.0-0.2
@@ -39,7 +146,7 @@ services:
 # Notes for Maintainers/Contributors who will change Line Numbers of Frontend & Query-Section. Please Update Line Numbers in `./scripts/commentLinesForSetup.sh` & `./CONTRIBUTING.md`
 
   query-service:
-    image: signoz/query-service:0.11.1
+    image: signoz/query-service:0.12.0
     container_name: query-service
     command: ["-config=/root/config/prometheus.yml"]
     # ports:
@@ -64,12 +171,10 @@ services:
       interval: 30s
       timeout: 5s
       retries: 3
-    depends_on:
-      clickhouse:
-        condition: service_healthy
+    <<: *clickhouse-depend
 
   frontend:
-    image: signoz/frontend:0.11.1
+    image: signoz/frontend:0.12.0
     container_name: frontend
     restart: on-failure
     depends_on:
@@ -81,7 +186,7 @@ services:
       - ../common/nginx-config.conf:/etc/nginx/conf.d/default.conf
 
   otel-collector:
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-config.yaml"]
     user: root # required for reading docker container logs
     volumes:
@@ -89,6 +194,7 @@ services:
       - /var/lib/docker/containers:/var/lib/docker/containers:ro
     environment:
       - OTEL_RESOURCE_ATTRIBUTES=host.name=signoz-host,os.type=linux
+      - DOCKER_MULTI_NODE_CLUSTER=false
     ports:
       # - "1777:1777"     # pprof extension
       - "4317:4317"     # OTLP gRPC receiver
@@ -102,12 +208,10 @@ services:
       # - "55678:55678"   # OpenCensus receiver
       # - "55679:55679"   # zPages extension
     restart: on-failure
-    depends_on:
-      clickhouse:
-        condition: service_healthy
+    <<: *clickhouse-depend
 
   otel-collector-metrics:
-    image: signoz/signoz-otel-collector:0.55.1
+    image: signoz/signoz-otel-collector:0.66.0
     command: ["--config=/etc/otel-collector-metrics-config.yaml"]
     volumes:
       - ./otel-collector-metrics-config.yaml:/etc/otel-collector-metrics-config.yaml
@@ -117,9 +221,7 @@ services:
     #   - "13133:13133"   # Health check extension
     #   - "55679:55679"   # zPages extension
     restart: on-failure
-    depends_on:
-      clickhouse:
-        condition: service_healthy
+    <<: *clickhouse-depend
 
   hotrod:
    image: jaegertracing/example-hotrod:1.30

deploy/docker/clickhouse-setup/otel-collector-config.yaml

@@ -47,7 +47,7 @@ receivers:
       # thrift_binary:
       #   endpoint: 0.0.0.0:6832
   hostmetrics:
-    collection_interval: 60s
+    collection_interval: 30s
     scrapers:
       cpu: {}
       load: {}
@@ -55,6 +55,16 @@ receivers:
       disk: {}
       filesystem: {}
       network: {}
+  prometheus:
+    config:
+      global:
+        scrape_interval: 60s
+      scrape_configs:
+        # otel-collector internal metrics
+        - job_name: otel-collector
+          static_configs:
+          - targets:
+            - localhost:8888
 
 processors:
   batch:
@@ -64,7 +74,7 @@ processors:
   signozspanmetrics/prometheus:
     metrics_exporter: prometheus
     latency_histogram_buckets: [100us, 1ms, 2ms, 6ms, 10ms, 50ms, 100ms, 250ms, 500ms, 1000ms, 1400ms, 2000ms, 5s, 10s, 20s, 40s, 60s ]
-    dimensions_cache_size: 10000
+    dimensions_cache_size: 100000
     dimensions:
       - name: service.namespace
         default: default
@@ -89,7 +99,6 @@ processors:
     # Using OTEL_RESOURCE_ATTRIBUTES envvar, env detector adds custom labels.
     detectors: [env, system] # include ec2 for AWS, gce for GCP and azure for Azure.
     timeout: 2s
-    override: false
 
 extensions:
   health_check:
@@ -102,6 +111,8 @@ extensions:
 exporters:
   clickhousetraces:
     datasource: tcp://clickhouse:9000/?database=signoz_traces
+    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
+
   clickhousemetricswrite:
     endpoint: tcp://clickhouse:9000/?database=signoz_metrics
     resource_to_telemetry_conversion:
@@ -112,6 +123,7 @@ exporters:
 
   clickhouselogsexporter:
     dsn: tcp://clickhouse:9000/
+    docker_multi_node_cluster: ${DOCKER_MULTI_NODE_CLUSTER}
     timeout: 5s
     sending_queue:
       queue_size: 100
@@ -138,8 +150,8 @@ service:
       receivers: [otlp]
       processors: [batch]
       exporters: [clickhousemetricswrite]
-    metrics/hostmetrics:
-      receivers: [hostmetrics]
+    metrics/generic:
+      receivers: [hostmetrics, prometheus]
       processors: [resourcedetection, batch]
       exporters: [clickhousemetricswrite]
     metrics/spanmetrics:

deploy/docker/clickhouse-setup/otel-collector-metrics-config.yaml

@@ -6,20 +6,14 @@ receivers:
   prometheus:
     config:
       scrape_configs:
-        # otel-collector internal metrics
-        - job_name: "otel-collector"
-          scrape_interval: 60s
-          static_configs:
-            - targets:
-              - otel-collector:8888
         # otel-collector-metrics internal metrics
-        - job_name: "otel-collector-metrics"
+        - job_name: otel-collector-metrics
           scrape_interval: 60s
           static_configs:
             - targets:
               - localhost:8888
         # SigNoz span metrics
-        - job_name: "signozspanmetrics-collector"
+        - job_name: signozspanmetrics-collector
           scrape_interval: 60s
           static_configs:
             - targets:

deploy/docker/common/nginx-config.conf

@@ -30,6 +30,8 @@ server {
 
     location /api {
         proxy_pass http://query-service:8080/api;
+        # connection will be closed if no data is read for 600s between successive read operations
+        proxy_read_timeout 600s; 
     }
 
     # redirect server error pages to the static page /50x.html

ee/query-service/app/api/api.go

@@ -93,6 +93,10 @@ func (ah *APIHandler) RegisterRoutes(router *mux.Router) {
 		baseapp.OpenAccess(ah.receiveSAML)).
 		Methods(http.MethodPost)
 
+	router.HandleFunc("/api/v1/complete/google",
+		baseapp.OpenAccess(ah.receiveGoogleAuth)).
+		Methods(http.MethodGet)
+		
 	router.HandleFunc("/api/v1/orgs/{orgId}/domains",
 		baseapp.AdminAccess(ah.listDomainsByOrg)).
 		Methods(http.MethodGet)
@@ -114,6 +118,9 @@ func (ah *APIHandler) RegisterRoutes(router *mux.Router) {
 	router.HandleFunc("/api/v1/invite/{token}", baseapp.OpenAccess(ah.getInvite)).Methods(http.MethodGet)
 	router.HandleFunc("/api/v1/register", baseapp.OpenAccess(ah.registerUser)).Methods(http.MethodPost)
 	router.HandleFunc("/api/v1/login", baseapp.OpenAccess(ah.loginUser)).Methods(http.MethodPost)
+	router.HandleFunc("/api/v1/traces/{traceId}", baseapp.ViewAccess(ah.searchTraces)).Methods(http.MethodGet)
+	router.HandleFunc("/api/v2/metrics/query_range", baseapp.ViewAccess(ah.queryRangeMetricsV2)).Methods(http.MethodPost)
+
 	ah.APIHandler.RegisterRoutes(router)
 
 }

ee/query-service/app/api/auth.go

@@ -8,9 +8,6 @@ import (
 	"io/ioutil"
 	"net/http"
 	"net/url"
-	"strings"
-
-	"github.com/google/uuid"
 	"github.com/gorilla/mux"
 	"go.signoz.io/signoz/ee/query-service/constants"
 	"go.signoz.io/signoz/ee/query-service/model"
@@ -184,114 +181,152 @@ func (ah *APIHandler) precheckLogin(w http.ResponseWriter, r *http.Request) {
 	ah.Respond(w, resp)
 }
 
-func (ah *APIHandler) receiveSAML(w http.ResponseWriter, r *http.Request) {
-	// this is the source url that initiated the login request
+func handleSsoError(w http.ResponseWriter, r *http.Request, redirectURL string) {
+	ssoError := []byte("Login failed. Please contact your system administrator")
+	dst := make([]byte, base64.StdEncoding.EncodedLen(len(ssoError)))
+	base64.StdEncoding.Encode(dst, ssoError)
+
+	http.Redirect(w, r, fmt.Sprintf("%s?ssoerror=%s", redirectURL, string(dst)), http.StatusSeeOther)
+}
+
+// receiveGoogleAuth completes google OAuth response and forwards a request
+// to front-end to sign user in 
+func (ah *APIHandler) receiveGoogleAuth(w http.ResponseWriter, r *http.Request) {
 	redirectUri := constants.GetDefaultSiteURL()
 	ctx := context.Background()
 
-	var apierr basemodel.BaseApiError
-
-	redirectOnError := func() {
-		ssoError := []byte("Login failed. Please contact your system administrator")
-		dst := make([]byte, base64.StdEncoding.EncodedLen(len(ssoError)))
-		base64.StdEncoding.Encode(dst, ssoError)
-
-		http.Redirect(w, r, fmt.Sprintf("%s?ssoerror=%s", redirectUri, string(dst)), http.StatusMovedPermanently)
-	}
-
 	if !ah.CheckFeature(model.SSO) {
-		zap.S().Errorf("[ReceiveSAML] sso requested but feature unavailable %s in org domain %s", model.SSO)
+		zap.S().Errorf("[receiveGoogleAuth] sso requested but feature unavailable %s in org domain %s", model.SSO)
 		http.Redirect(w, r, fmt.Sprintf("%s?ssoerror=%s", redirectUri, "feature unavailable, please upgrade your billing plan to access this feature"), http.StatusMovedPermanently)
 		return
 	}
 
-	err := r.ParseForm()
-	if err != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to process response - invalid response from IDP", err, r)
-		redirectOnError()
+	q := r.URL.Query()
+	if errType := q.Get("error"); errType != "" {
+		zap.S().Errorf("[receiveGoogleAuth] failed to login with google auth", q.Get("error_description"))
+		http.Redirect(w, r, fmt.Sprintf("%s?ssoerror=%s", redirectUri, "failed to login through SSO "), http.StatusMovedPermanently)
 		return
 	}
 
-	// the relay state is sent when a login request is submitted to
-	// Idp.
-	relayState := r.FormValue("RelayState")
-	zap.S().Debug("[ReceiveML] relay state", zap.String("relayState", relayState))
+	relayState := q.Get("state")
+	zap.S().Debug("[receiveGoogleAuth] relay state received", zap.String("state", relayState))
 
 	parsedState, err := url.Parse(relayState)
 	if err != nil || relayState == "" {
-		zap.S().Errorf("[ReceiveSAML] failed to process response - invalid response from IDP", err, r)
-		redirectOnError()
+		zap.S().Errorf("[receiveGoogleAuth] failed to process response - invalid response from IDP", err, r)
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
 	// upgrade redirect url from the relay state for better accuracy
 	redirectUri = fmt.Sprintf("%s://%s%s", parsedState.Scheme, parsedState.Host, "/login")
 
-	// derive domain id from relay state now
-	var domainIdStr string
-	for k, v := range parsedState.Query() {
-		if k == "domainId" && len(v) > 0 {
-			domainIdStr = strings.Replace(v[0], ":", "-", -1)
-		}
-	}
-
-	domainId, err := uuid.Parse(domainIdStr)
+	// fetch domain by parsing relay state. 
+	domain, err := ah.AppDao().GetDomainFromSsoResponse(ctx, parsedState)
 	if err != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to process request- failed to parse domain id ifrom relay", zap.Error(err))
-		redirectOnError()
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
-	domain, apierr := ah.AppDao().GetDomain(ctx, domainId)
-	if (apierr != nil) || domain == nil {
-		zap.S().Errorf("[ReceiveSAML] failed to process request- invalid domain", domainIdStr, zap.Error(apierr))
-		redirectOnError()
+	// now that we have domain, use domain to fetch sso settings. 
+	// prepare google callback handler using parsedState - 
+	// which contains redirect URL (front-end endpoint)
+	callbackHandler, err := domain.PrepareGoogleOAuthProvider(parsedState)
+
+	identity, err := callbackHandler.HandleCallback(r)
+	if err != nil {
+		zap.S().Errorf("[receiveGoogleAuth] failed to process HandleCallback ", domain.String(), zap.Error(err))
+		handleSsoError(w, r, redirectUri)
+		return
+	}
+	
+	nextPage, err := ah.AppDao().PrepareSsoRedirect(ctx, redirectUri, identity.Email)
+	if err != nil {
+		zap.S().Errorf("[receiveGoogleAuth] failed to generate redirect URI after successful login ", domain.String(), zap.Error(err))
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
+	http.Redirect(w, r, nextPage, http.StatusSeeOther)
+}
+
+
+
+// receiveSAML completes a SAML request and gets user logged in
+func (ah *APIHandler) receiveSAML(w http.ResponseWriter, r *http.Request) {
+	// this is the source url that initiated the login request
+	redirectUri := constants.GetDefaultSiteURL()
+	ctx := context.Background()
+
+
+	if !ah.CheckFeature(model.SSO) {
+		zap.S().Errorf("[receiveSAML] sso requested but feature unavailable %s in org domain %s", model.SSO)
+		http.Redirect(w, r, fmt.Sprintf("%s?ssoerror=%s", redirectUri, "feature unavailable, please upgrade your billing plan to access this feature"), http.StatusMovedPermanently)
+		return
+	}
+
+	err := r.ParseForm()
+	if err != nil {
+		zap.S().Errorf("[receiveSAML] failed to process response - invalid response from IDP", err, r)
+		handleSsoError(w, r, redirectUri)
+		return
+	}
+
+	// the relay state is sent when a login request is submitted to
+	// Idp.
+	relayState := r.FormValue("RelayState")
+	zap.S().Debug("[receiveML] relay state", zap.String("relayState", relayState))
+
+	parsedState, err := url.Parse(relayState)
+	if err != nil || relayState == "" {
+		zap.S().Errorf("[receiveSAML] failed to process response - invalid response from IDP", err, r)
+		handleSsoError(w, r, redirectUri)
+		return
+	}
+
+	// upgrade redirect url from the relay state for better accuracy
+	redirectUri = fmt.Sprintf("%s://%s%s", parsedState.Scheme, parsedState.Host, "/login")
+
+	// fetch domain by parsing relay state. 
+	domain, err := ah.AppDao().GetDomainFromSsoResponse(ctx, parsedState)
+	if err != nil {
+		handleSsoError(w, r, redirectUri)
+		return
+	}
+	
 	sp, err := domain.PrepareSamlRequest(parsedState)
 	if err != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to prepare saml request for domain (%s): %v", domainId, err)
-		redirectOnError()
+		zap.S().Errorf("[receiveSAML] failed to prepare saml request for domain (%s): %v", domain.String(), err)
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
 	assertionInfo, err := sp.RetrieveAssertionInfo(r.FormValue("SAMLResponse"))
 	if err != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to retrieve assertion info from  saml response for organization (%s): %v", domainId, err)
-		redirectOnError()
+		zap.S().Errorf("[receiveSAML] failed to retrieve assertion info from  saml response for organization (%s): %v", domain.String(), err)
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
 	if assertionInfo.WarningInfo.InvalidTime {
-		zap.S().Errorf("[ReceiveSAML] expired saml response for organization (%s): %v", domainId, err)
-		redirectOnError()
+		zap.S().Errorf("[receiveSAML] expired saml response for organization (%s): %v", domain.String(), err)
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
 	email := assertionInfo.NameID
-
-	// user email found, now start preparing jwt response
-	userPayload, baseapierr := ah.AppDao().GetUserByEmail(ctx, email)
-	if baseapierr != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to find or register a new user for email %s and org %s", email, domainId, zap.Error(baseapierr.Err))
-		redirectOnError()
+	if email == "" {
+		zap.S().Errorf("[receiveSAML] invalid email in the SSO response (%s)", domain.String())
+		handleSsoError(w, r, redirectUri)
 		return
 	}
 
-	tokenStore, err := baseauth.GenerateJWTForUser(&userPayload.User)
+	nextPage, err := ah.AppDao().PrepareSsoRedirect(ctx, redirectUri, email)
 	if err != nil {
-		zap.S().Errorf("[ReceiveSAML] failed to generate access token for email %s and org %s", email, domainId, zap.Error(err))
-		redirectOnError()
+		zap.S().Errorf("[receiveSAML] failed to generate redirect URI after successful login ", domain.String(), zap.Error(err))
+		handleSsoError(w, r, redirectUri)
 		return
 	}
-
-	userID := userPayload.User.Id
-	nextPage := fmt.Sprintf("%s?jwt=%s&usr=%s&refreshjwt=%s",
-		redirectUri,
-		tokenStore.AccessJwt,
-		userID,
-		tokenStore.RefreshJwt)
-
-	http.Redirect(w, r, nextPage, http.StatusMovedPermanently)
+	
+	http.Redirect(w, r, nextPage, http.StatusSeeOther)
 }

ee/query-service/app/api/metrics.go

@@ -0,0 +1,236 @@
+package api
+
+import (
+	"bytes"
+	"fmt"
+	"net/http"
+	"sync"
+	"text/template"
+	"time"
+
+	"go.signoz.io/signoz/pkg/query-service/app/metrics"
+	"go.signoz.io/signoz/pkg/query-service/app/parser"
+	"go.signoz.io/signoz/pkg/query-service/constants"
+	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	querytemplate "go.signoz.io/signoz/pkg/query-service/utils/queryTemplate"
+	"go.uber.org/zap"
+)
+
+func (ah *APIHandler) queryRangeMetricsV2(w http.ResponseWriter, r *http.Request) {
+	if !ah.CheckFeature(basemodel.CustomMetricsFunction) {
+		zap.S().Info("CustomMetricsFunction feature is not enabled in this plan")
+		ah.APIHandler.QueryRangeMetricsV2(w, r)
+		return
+	}
+	metricsQueryRangeParams, apiErrorObj := parser.ParseMetricQueryRangeParams(r)
+
+	if apiErrorObj != nil {
+		zap.S().Errorf(apiErrorObj.Err.Error())
+		RespondError(w, apiErrorObj, nil)
+		return
+	}
+
+	// prometheus instant query needs same timestamp
+	if metricsQueryRangeParams.CompositeMetricQuery.PanelType == basemodel.QUERY_VALUE &&
+		metricsQueryRangeParams.CompositeMetricQuery.QueryType == basemodel.PROM {
+		metricsQueryRangeParams.Start = metricsQueryRangeParams.End
+	}
+
+	// round up the end to nearest multiple
+	if metricsQueryRangeParams.CompositeMetricQuery.QueryType == basemodel.QUERY_BUILDER {
+		end := (metricsQueryRangeParams.End) / 1000
+		step := metricsQueryRangeParams.Step
+		metricsQueryRangeParams.End = (end / step * step) * 1000
+	}
+
+	type channelResult struct {
+		Series    []*basemodel.Series
+		TableName string
+		Err       error
+		Name      string
+		Query     string
+	}
+
+	execClickHouseQueries := func(queries map[string]string) ([]*basemodel.Series, []string, error, map[string]string) {
+		var seriesList []*basemodel.Series
+		var tableName []string
+		ch := make(chan channelResult, len(queries))
+		var wg sync.WaitGroup
+
+		for name, query := range queries {
+			wg.Add(1)
+			go func(name, query string) {
+				defer wg.Done()
+				seriesList, tableName, err := ah.opts.DataConnector.GetMetricResultEE(r.Context(), query)
+				for _, series := range seriesList {
+					series.QueryName = name
+				}
+
+				if err != nil {
+					ch <- channelResult{Err: fmt.Errorf("error in query-%s: %v", name, err), Name: name, Query: query}
+					return
+				}
+				ch <- channelResult{Series: seriesList, TableName: tableName}
+			}(name, query)
+		}
+
+		wg.Wait()
+		close(ch)
+
+		var errs []error
+		errQuriesByName := make(map[string]string)
+		// read values from the channel
+		for r := range ch {
+			if r.Err != nil {
+				errs = append(errs, r.Err)
+				errQuriesByName[r.Name] = r.Query
+				continue
+			}
+			seriesList = append(seriesList, r.Series...)
+			tableName = append(tableName, r.TableName)
+		}
+		if len(errs) != 0 {
+			return nil, nil, fmt.Errorf("encountered multiple errors: %s", metrics.FormatErrs(errs, "\n")), errQuriesByName
+		}
+		return seriesList, tableName, nil, nil
+	}
+
+	execPromQueries := func(metricsQueryRangeParams *basemodel.QueryRangeParamsV2) ([]*basemodel.Series, error, map[string]string) {
+		var seriesList []*basemodel.Series
+		ch := make(chan channelResult, len(metricsQueryRangeParams.CompositeMetricQuery.PromQueries))
+		var wg sync.WaitGroup
+
+		for name, query := range metricsQueryRangeParams.CompositeMetricQuery.PromQueries {
+			if query.Disabled {
+				continue
+			}
+			wg.Add(1)
+			go func(name string, query *basemodel.PromQuery) {
+				var seriesList []*basemodel.Series
+				defer wg.Done()
+				tmpl := template.New("promql-query")
+				tmpl, tmplErr := tmpl.Parse(query.Query)
+				if tmplErr != nil {
+					ch <- channelResult{Err: fmt.Errorf("error in parsing query-%s: %v", name, tmplErr), Name: name, Query: query.Query}
+					return
+				}
+				var queryBuf bytes.Buffer
+				tmplErr = tmpl.Execute(&queryBuf, metricsQueryRangeParams.Variables)
+				if tmplErr != nil {
+					ch <- channelResult{Err: fmt.Errorf("error in parsing query-%s: %v", name, tmplErr), Name: name, Query: query.Query}
+					return
+				}
+				query.Query = queryBuf.String()
+				queryModel := basemodel.QueryRangeParams{
+					Start: time.UnixMilli(metricsQueryRangeParams.Start),
+					End:   time.UnixMilli(metricsQueryRangeParams.End),
+					Step:  time.Duration(metricsQueryRangeParams.Step * int64(time.Second)),
+					Query: query.Query,
+				}
+				promResult, _, err := ah.opts.DataConnector.GetQueryRangeResult(r.Context(), &queryModel)
+				if err != nil {
+					ch <- channelResult{Err: fmt.Errorf("error in query-%s: %v", name, err), Name: name, Query: query.Query}
+					return
+				}
+				matrix, _ := promResult.Matrix()
+				for _, v := range matrix {
+					var s basemodel.Series
+					s.QueryName = name
+					s.Labels = v.Metric.Copy().Map()
+					for _, p := range v.Points {
+						s.Points = append(s.Points, basemodel.MetricPoint{Timestamp: p.T, Value: p.V})
+					}
+					seriesList = append(seriesList, &s)
+				}
+				ch <- channelResult{Series: seriesList}
+			}(name, query)
+		}
+
+		wg.Wait()
+		close(ch)
+
+		var errs []error
+		errQuriesByName := make(map[string]string)
+		// read values from the channel
+		for r := range ch {
+			if r.Err != nil {
+				errs = append(errs, r.Err)
+				errQuriesByName[r.Name] = r.Query
+				continue
+			}
+			seriesList = append(seriesList, r.Series...)
+		}
+		if len(errs) != 0 {
+			return nil, fmt.Errorf("encountered multiple errors: %s", metrics.FormatErrs(errs, "\n")), errQuriesByName
+		}
+		return seriesList, nil, nil
+	}
+
+	var seriesList []*basemodel.Series
+	var tableName []string
+	var err error
+	var errQuriesByName map[string]string
+	switch metricsQueryRangeParams.CompositeMetricQuery.QueryType {
+	case basemodel.QUERY_BUILDER:
+		runQueries := metrics.PrepareBuilderMetricQueries(metricsQueryRangeParams, constants.SIGNOZ_TIMESERIES_TABLENAME)
+		if runQueries.Err != nil {
+			RespondError(w, &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: runQueries.Err}, nil)
+			return
+		}
+		seriesList, tableName, err, errQuriesByName = execClickHouseQueries(runQueries.Queries)
+
+	case basemodel.CLICKHOUSE:
+		queries := make(map[string]string)
+
+		for name, chQuery := range metricsQueryRangeParams.CompositeMetricQuery.ClickHouseQueries {
+			if chQuery.Disabled {
+				continue
+			}
+			tmpl := template.New("clickhouse-query")
+			tmpl, err := tmpl.Parse(chQuery.Query)
+			if err != nil {
+				RespondError(w, &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: err}, nil)
+				return
+			}
+			var query bytes.Buffer
+
+			// replace go template variables
+			querytemplate.AssignReservedVars(metricsQueryRangeParams)
+
+			err = tmpl.Execute(&query, metricsQueryRangeParams.Variables)
+			if err != nil {
+				RespondError(w, &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: err}, nil)
+				return
+			}
+			queries[name] = query.String()
+		}
+		seriesList, tableName, err, errQuriesByName = execClickHouseQueries(queries)
+	case basemodel.PROM:
+		seriesList, err, errQuriesByName = execPromQueries(metricsQueryRangeParams)
+	default:
+		err = fmt.Errorf("invalid query type")
+		RespondError(w, &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: err}, errQuriesByName)
+		return
+	}
+
+	if err != nil {
+		apiErrObj := &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: err}
+		RespondError(w, apiErrObj, errQuriesByName)
+		return
+	}
+	if metricsQueryRangeParams.CompositeMetricQuery.PanelType == basemodel.QUERY_VALUE &&
+		len(seriesList) > 1 &&
+		(metricsQueryRangeParams.CompositeMetricQuery.QueryType == basemodel.QUERY_BUILDER ||
+			metricsQueryRangeParams.CompositeMetricQuery.QueryType == basemodel.CLICKHOUSE) {
+		RespondError(w, &basemodel.ApiError{Typ: basemodel.ErrorBadData, Err: fmt.Errorf("invalid: query resulted in more than one series for value type")}, nil)
+		return
+	}
+
+	type ResponseFormat struct {
+		ResultType string              `json:"resultType"`
+		Result     []*basemodel.Series `json:"result"`
+		TableName  []string            `json:"tableName"`
+	}
+	resp := ResponseFormat{ResultType: "matrix", Result: seriesList, TableName: tableName}
+	ah.Respond(w, resp)
+}

ee/query-service/app/api/traces.go

@@ -0,0 +1,39 @@
+package api
+
+import (
+	"net/http"
+	"strconv"
+
+	"go.signoz.io/signoz/ee/query-service/app/db"
+	"go.signoz.io/signoz/ee/query-service/constants"
+	"go.signoz.io/signoz/ee/query-service/model"
+	baseapp "go.signoz.io/signoz/pkg/query-service/app"
+	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	"go.uber.org/zap"
+)
+
+func (ah *APIHandler) searchTraces(w http.ResponseWriter, r *http.Request) {
+
+	if !ah.CheckFeature(basemodel.SmartTraceDetail) {
+		zap.S().Info("SmartTraceDetail feature is not enabled in this plan")
+		ah.APIHandler.SearchTraces(w, r)
+		return
+	}
+	traceId, spanId, levelUpInt, levelDownInt, err := baseapp.ParseSearchTracesParams(r)
+	if err != nil {
+		RespondError(w, &model.ApiError{Typ: model.ErrorBadData, Err: err}, "Error reading params")
+		return
+	}
+	spanLimit, err := strconv.Atoi(constants.SpanLimitStr)
+	if err != nil {
+		zap.S().Error("Error during strconv.Atoi() on SPAN_LIMIT env variable: ", err)
+		return
+	}
+	result, err := ah.opts.DataConnector.SearchTraces(r.Context(), traceId, spanId, levelUpInt, levelDownInt, spanLimit, db.SmartTraceAlgorithm)
+	if ah.HandleError(w, err, http.StatusBadRequest) {
+		return
+	}
+
+	ah.WriteJSON(w, r, result)
+
+}

ee/query-service/app/db/metrics.go

@@ -0,0 +1,401 @@
+package db
+
+import (
+	"context"
+	"crypto/md5"
+	"encoding/json"
+	"fmt"
+	"reflect"
+	"regexp"
+	"sort"
+	"strings"
+	"time"
+
+	"go.signoz.io/signoz/ee/query-service/model"
+	baseconst "go.signoz.io/signoz/pkg/query-service/constants"
+	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	"go.signoz.io/signoz/pkg/query-service/utils"
+	"go.uber.org/zap"
+)
+
+// GetMetricResultEE runs the query and returns list of time series
+func (r *ClickhouseReader) GetMetricResultEE(ctx context.Context, query string) ([]*basemodel.Series, string, error) {
+
+	defer utils.Elapsed("GetMetricResult")()
+	zap.S().Infof("Executing metric result query: %s", query)
+
+	var hash string
+	// If getSubTreeSpans function is used in the clickhouse query
+	if strings.Index(query, "getSubTreeSpans(") != -1 {
+		var err error
+		query, hash, err = r.getSubTreeSpansCustomFunction(ctx, query, hash)
+		if err == fmt.Errorf("No spans found for the given query") {
+			return nil, "", nil
+		}
+		if err != nil {
+			return nil, "", err
+		}
+	}
+
+	rows, err := r.conn.Query(ctx, query)
+	zap.S().Debug(query)
+	if err != nil {
+		zap.S().Debug("Error in processing query: ", err)
+		return nil, "", fmt.Errorf("error in processing query")
+	}
+
+	var (
+		columnTypes = rows.ColumnTypes()
+		columnNames = rows.Columns()
+		vars        = make([]interface{}, len(columnTypes))
+	)
+	for i := range columnTypes {
+		vars[i] = reflect.New(columnTypes[i].ScanType()).Interface()
+	}
+	// when group by is applied, each combination of cartesian product
+	// of attributes is separate series. each item in metricPointsMap
+	// represent a unique series.
+	metricPointsMap := make(map[string][]basemodel.MetricPoint)
+	// attribute key-value pairs for each group selection
+	attributesMap := make(map[string]map[string]string)
+
+	defer rows.Close()
+	for rows.Next() {
+		if err := rows.Scan(vars...); err != nil {
+			return nil, "", err
+		}
+		var groupBy []string
+		var metricPoint basemodel.MetricPoint
+		groupAttributes := make(map[string]string)
+		// Assuming that the end result row contains a timestamp, value and option labels
+		// Label key and value are both strings.
+		for idx, v := range vars {
+			colName := columnNames[idx]
+			switch v := v.(type) {
+			case *string:
+				// special case for returning all labels
+				if colName == "fullLabels" {
+					var metric map[string]string
+					err := json.Unmarshal([]byte(*v), &metric)
+					if err != nil {
+						return nil, "", err
+					}
+					for key, val := range metric {
+						groupBy = append(groupBy, val)
+						groupAttributes[key] = val
+					}
+				} else {
+					groupBy = append(groupBy, *v)
+					groupAttributes[colName] = *v
+				}
+			case *time.Time:
+				metricPoint.Timestamp = v.UnixMilli()
+			case *float64:
+				metricPoint.Value = *v
+			case **float64:
+				// ch seems to return this type when column is derived from
+				// SELECT count(*)/ SELECT count(*)
+				floatVal := *v
+				if floatVal != nil {
+					metricPoint.Value = *floatVal
+				}
+			case *float32:
+				float32Val := float32(*v)
+				metricPoint.Value = float64(float32Val)
+			case *uint8, *uint64, *uint16, *uint32:
+				if _, ok := baseconst.ReservedColumnTargetAliases[colName]; ok {
+					metricPoint.Value = float64(reflect.ValueOf(v).Elem().Uint())
+				} else {
+					groupBy = append(groupBy, fmt.Sprintf("%v", reflect.ValueOf(v).Elem().Uint()))
+					groupAttributes[colName] = fmt.Sprintf("%v", reflect.ValueOf(v).Elem().Uint())
+				}
+			case *int8, *int16, *int32, *int64:
+				if _, ok := baseconst.ReservedColumnTargetAliases[colName]; ok {
+					metricPoint.Value = float64(reflect.ValueOf(v).Elem().Int())
+				} else {
+					groupBy = append(groupBy, fmt.Sprintf("%v", reflect.ValueOf(v).Elem().Int()))
+					groupAttributes[colName] = fmt.Sprintf("%v", reflect.ValueOf(v).Elem().Int())
+				}
+			default:
+				zap.S().Errorf("invalid var found in metric builder query result", v, colName)
+			}
+		}
+		sort.Strings(groupBy)
+		key := strings.Join(groupBy, "")
+		attributesMap[key] = groupAttributes
+		metricPointsMap[key] = append(metricPointsMap[key], metricPoint)
+	}
+
+	var seriesList []*basemodel.Series
+	for key := range metricPointsMap {
+		points := metricPointsMap[key]
+		// first point in each series could be invalid since the
+		// aggregations are applied with point from prev series
+		if len(points) != 0 && len(points) > 1 {
+			points = points[1:]
+		}
+		attributes := attributesMap[key]
+		series := basemodel.Series{Labels: attributes, Points: points}
+		seriesList = append(seriesList, &series)
+	}
+	// err = r.conn.Exec(ctx, "DROP TEMPORARY TABLE IF EXISTS getSubTreeSpans"+hash)
+	// if err != nil {
+	// 	zap.S().Error("Error in dropping temporary table: ", err)
+	// 	return nil, err
+	// }
+	if hash == "" {
+		return seriesList, hash, nil
+	} else {
+		return seriesList, "getSubTreeSpans" + hash, nil
+	}
+}
+
+func (r *ClickhouseReader) getSubTreeSpansCustomFunction(ctx context.Context, query string, hash string) (string, string, error) {
+
+	zap.S().Debugf("Executing getSubTreeSpans function")
+
+	// str1 := `select fromUnixTimestamp64Milli(intDiv( toUnixTimestamp64Milli ( timestamp ), 100) * 100) AS interval, toFloat64(count()) as count from (select timestamp, spanId, parentSpanId, durationNano from getSubTreeSpans(select * from signoz_traces.signoz_index_v2 where serviceName='frontend' and name='/driver.DriverService/FindNearest' and  traceID='00000000000000004b0a863cb5ed7681') where name='FindDriverIDs' group by interval order by interval asc;`
+
+	// process the query to fetch subTree query
+	var subtreeInput string
+	query, subtreeInput, hash = processQuery(query, hash)
+
+	err := r.conn.Exec(ctx, "DROP TABLE IF EXISTS getSubTreeSpans"+hash)
+	if err != nil {
+		zap.S().Error("Error in dropping temporary table: ", err)
+		return query, hash, err
+	}
+
+	// Create temporary table to store the getSubTreeSpans() results
+	zap.S().Debugf("Creating temporary table getSubTreeSpans%s", hash)
+	err = r.conn.Exec(ctx, "CREATE TABLE IF NOT EXISTS "+"getSubTreeSpans"+hash+" (timestamp DateTime64(9) CODEC(DoubleDelta, LZ4), traceID FixedString(32) CODEC(ZSTD(1)), spanID String CODEC(ZSTD(1)), parentSpanID String CODEC(ZSTD(1)), rootSpanID String CODEC(ZSTD(1)), serviceName LowCardinality(String) CODEC(ZSTD(1)), name LowCardinality(String) CODEC(ZSTD(1)), rootName LowCardinality(String) CODEC(ZSTD(1)), durationNano UInt64 CODEC(T64, ZSTD(1)), kind Int8 CODEC(T64, ZSTD(1)), tagMap Map(LowCardinality(String), String) CODEC(ZSTD(1)), events Array(String) CODEC(ZSTD(2))) ENGINE = MergeTree() ORDER BY (timestamp)")
+	if err != nil {
+		zap.S().Error("Error in creating temporary table: ", err)
+		return query, hash, err
+	}
+
+	var getSpansSubQueryDBResponses []model.GetSpansSubQueryDBResponse
+	getSpansSubQuery := subtreeInput
+	// Execute the subTree query
+	zap.S().Debugf("Executing subTree query: %s", getSpansSubQuery)
+	err = r.conn.Select(ctx, &getSpansSubQueryDBResponses, getSpansSubQuery)
+
+	// zap.S().Info(getSpansSubQuery)
+
+	if err != nil {
+		zap.S().Debug("Error in processing sql query: ", err)
+		return query, hash, fmt.Errorf("Error in processing sql query")
+	}
+
+	var searchScanResponses []basemodel.SearchSpanDBResponseItem
+
+	// TODO : @ankit: I think the algorithm does not need to assume that subtrees are from the same TraceID. We can take this as an improvement later.
+	// Fetch all the spans from of same TraceID so that we can build subtree
+	modelQuery := fmt.Sprintf("SELECT timestamp, traceID, model FROM %s.%s WHERE traceID=$1", r.TraceDB, r.SpansTable)
+
+	if len(getSpansSubQueryDBResponses) == 0 {
+		return query, hash, fmt.Errorf("No spans found for the given query")
+	}
+	zap.S().Debugf("Executing query to fetch all the spans from the same TraceID: %s", modelQuery)
+	err = r.conn.Select(ctx, &searchScanResponses, modelQuery, getSpansSubQueryDBResponses[0].TraceID)
+
+	if err != nil {
+		zap.S().Debug("Error in processing sql query: ", err)
+		return query, hash, fmt.Errorf("Error in processing sql query")
+	}
+
+	// Process model to fetch the spans
+	zap.S().Debugf("Processing model to fetch the spans")
+	searchSpanResponses := []basemodel.SearchSpanResponseItem{}
+	for _, item := range searchScanResponses {
+		var jsonItem basemodel.SearchSpanResponseItem
+		json.Unmarshal([]byte(item.Model), &jsonItem)
+		jsonItem.TimeUnixNano = uint64(item.Timestamp.UnixNano())
+		if jsonItem.Events == nil {
+			jsonItem.Events = []string{}
+		}
+		searchSpanResponses = append(searchSpanResponses, jsonItem)
+	}
+	// Build the subtree and store all the subtree spans in temporary table getSubTreeSpans+hash
+	// Use map to store pointer to the spans to avoid duplicates and save memory
+	zap.S().Debugf("Building the subtree to store all the subtree spans in temporary table getSubTreeSpans%s", hash)
+
+	treeSearchResponse, err := getSubTreeAlgorithm(searchSpanResponses, getSpansSubQueryDBResponses)
+	if err != nil {
+		zap.S().Error("Error in getSubTreeAlgorithm function: ", err)
+		return query, hash, err
+	}
+	zap.S().Debugf("Preparing batch to store subtree spans in temporary table getSubTreeSpans%s", hash)
+	statement, err := r.conn.PrepareBatch(context.Background(), fmt.Sprintf("INSERT INTO getSubTreeSpans"+hash))
+	if err != nil {
+		zap.S().Error("Error in preparing batch statement: ", err)
+		return query, hash, err
+	}
+	for _, span := range treeSearchResponse {
+		var parentID string
+		if len(span.References) > 0 && span.References[0].RefType == "CHILD_OF" {
+			parentID = span.References[0].SpanId
+		}
+		err = statement.Append(
+			time.Unix(0, int64(span.TimeUnixNano)),
+			span.TraceID,
+			span.SpanID,
+			parentID,
+			span.RootSpanID,
+			span.ServiceName,
+			span.Name,
+			span.RootName,
+			uint64(span.DurationNano),
+			int8(span.Kind),
+			span.TagMap,
+			span.Events,
+		)
+		if err != nil {
+			zap.S().Debug("Error in processing sql query: ", err)
+			return query, hash, err
+		}
+	}
+	zap.S().Debugf("Inserting the subtree spans in temporary table getSubTreeSpans%s", hash)
+	err = statement.Send()
+	if err != nil {
+		zap.S().Error("Error in sending statement: ", err)
+		return query, hash, err
+	}
+	return query, hash, nil
+}
+
+func processQuery(query string, hash string) (string, string, string) {
+	re3 := regexp.MustCompile(`getSubTreeSpans`)
+
+	submatchall3 := re3.FindAllStringIndex(query, -1)
+	getSubtreeSpansMatchIndex := submatchall3[0][1]
+
+	query2countParenthesis := query[getSubtreeSpansMatchIndex:]
+
+	sqlCompleteIndex := 0
+	countParenthesisImbalance := 0
+	for i, char := range query2countParenthesis {
+
+		if string(char) == "(" {
+			countParenthesisImbalance += 1
+		}
+		if string(char) == ")" {
+			countParenthesisImbalance -= 1
+		}
+		if countParenthesisImbalance == 0 {
+			sqlCompleteIndex = i
+			break
+		}
+	}
+	subtreeInput := query2countParenthesis[1:sqlCompleteIndex]
+
+	// hash the subtreeInput
+	hmd5 := md5.Sum([]byte(subtreeInput))
+	hash = fmt.Sprintf("%x", hmd5)
+
+	// Reformat the query to use the getSubTreeSpans function
+	query = query[:getSubtreeSpansMatchIndex] + hash + " " + query2countParenthesis[sqlCompleteIndex+1:]
+	return query, subtreeInput, hash
+}
+
+// getSubTreeAlgorithm is an algorithm to build the subtrees of the spans and return the list of spans
+func getSubTreeAlgorithm(payload []basemodel.SearchSpanResponseItem, getSpansSubQueryDBResponses []model.GetSpansSubQueryDBResponse) (map[string]*basemodel.SearchSpanResponseItem, error) {
+
+	var spans []*model.SpanForTraceDetails
+	for _, spanItem := range payload {
+		var parentID string
+		if len(spanItem.References) > 0 && spanItem.References[0].RefType == "CHILD_OF" {
+			parentID = spanItem.References[0].SpanId
+		}
+		span := &model.SpanForTraceDetails{
+			TimeUnixNano: spanItem.TimeUnixNano,
+			SpanID:       spanItem.SpanID,
+			TraceID:      spanItem.TraceID,
+			ServiceName:  spanItem.ServiceName,
+			Name:         spanItem.Name,
+			Kind:         spanItem.Kind,
+			DurationNano: spanItem.DurationNano,
+			TagMap:       spanItem.TagMap,
+			ParentID:     parentID,
+			Events:       spanItem.Events,
+			HasError:     spanItem.HasError,
+		}
+		spans = append(spans, span)
+	}
+
+	zap.S().Debug("Building Tree")
+	roots, err := buildSpanTrees(&spans)
+	if err != nil {
+		return nil, err
+	}
+	searchSpansResult := make(map[string]*basemodel.SearchSpanResponseItem)
+	// Every span which was fetched from getSubTree Input SQL query is considered root
+	// For each root, get the subtree spans
+	for _, getSpansSubQueryDBResponse := range getSpansSubQueryDBResponses {
+		targetSpan := &model.SpanForTraceDetails{}
+		// zap.S().Debug("Building tree for span id: " + getSpansSubQueryDBResponse.SpanID + " " + strconv.Itoa(i+1) + " of " + strconv.Itoa(len(getSpansSubQueryDBResponses)))
+		// Search target span object in the tree
+		for _, root := range roots {
+			targetSpan, err = breadthFirstSearch(root, getSpansSubQueryDBResponse.SpanID)
+			if targetSpan != nil {
+				break
+			}
+			if err != nil {
+				zap.S().Error("Error during BreadthFirstSearch(): ", err)
+				return nil, err
+			}
+		}
+		if targetSpan == nil {
+			return nil, nil
+		}
+		// Build subtree for the target span
+		// Mark the target span as root by setting parent ID as empty string
+		targetSpan.ParentID = ""
+		preParents := []*model.SpanForTraceDetails{targetSpan}
+		children := []*model.SpanForTraceDetails{}
+
+		// Get the subtree child spans
+		for i := 0; len(preParents) != 0; i++ {
+			parents := []*model.SpanForTraceDetails{}
+			for _, parent := range preParents {
+				children = append(children, parent.Children...)
+				parents = append(parents, parent.Children...)
+			}
+			preParents = parents
+		}
+
+		resultSpans := children
+		// Add the target span to the result spans
+		resultSpans = append(resultSpans, targetSpan)
+
+		for _, item := range resultSpans {
+			references := []basemodel.OtelSpanRef{
+				{
+					TraceId: item.TraceID,
+					SpanId:  item.ParentID,
+					RefType: "CHILD_OF",
+				},
+			}
+
+			if item.Events == nil {
+				item.Events = []string{}
+			}
+			searchSpansResult[item.SpanID] = &basemodel.SearchSpanResponseItem{
+				TimeUnixNano: item.TimeUnixNano,
+				SpanID:       item.SpanID,
+				TraceID:      item.TraceID,
+				ServiceName:  item.ServiceName,
+				Name:         item.Name,
+				Kind:         item.Kind,
+				References:   references,
+				DurationNano: item.DurationNano,
+				TagMap:       item.TagMap,
+				Events:       item.Events,
+				HasError:     item.HasError,
+				RootSpanID:   getSpansSubQueryDBResponse.SpanID,
+				RootName:     targetSpan.Name,
+			}
+		}
+	}
+	return searchSpansResult, nil
+}

ee/query-service/app/db/reader.go

@@ -6,6 +6,7 @@ import (
 	"github.com/jmoiron/sqlx"
 
 	basechr "go.signoz.io/signoz/pkg/query-service/app/clickhouseReader"
+	"go.signoz.io/signoz/pkg/query-service/interfaces"
 )
 
 type ClickhouseReader struct {
@@ -14,8 +15,8 @@ type ClickhouseReader struct {
 	*basechr.ClickHouseReader
 }
 
-func NewDataConnector(localDB *sqlx.DB, promConfigPath string) *ClickhouseReader {
-	ch := basechr.NewReader(localDB, promConfigPath)
+func NewDataConnector(localDB *sqlx.DB, promConfigPath string, lm interfaces.FeatureLookup) *ClickhouseReader {
+	ch := basechr.NewReader(localDB, promConfigPath, lm)
 	return &ClickhouseReader{
 		conn:             ch.GetConn(),
 		appdb:            localDB,

ee/query-service/app/db/trace.go

@@ -0,0 +1,222 @@
+package db
+
+import (
+	"errors"
+	"strconv"
+
+	"go.signoz.io/signoz/ee/query-service/model"
+	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	"go.uber.org/zap"
+)
+
+// SmartTraceAlgorithm is an algorithm to find the target span and build a tree of spans around it with the given levelUp and levelDown parameters and the given spanLimit
+func SmartTraceAlgorithm(payload []basemodel.SearchSpanResponseItem, targetSpanId string, levelUp int, levelDown int, spanLimit int) ([]basemodel.SearchSpansResult, error) {
+	var spans []*model.SpanForTraceDetails
+
+	// Build a slice of spans from the payload
+	for _, spanItem := range payload {
+		var parentID string
+		if len(spanItem.References) > 0 && spanItem.References[0].RefType == "CHILD_OF" {
+			parentID = spanItem.References[0].SpanId
+		}
+		span := &model.SpanForTraceDetails{
+			TimeUnixNano: spanItem.TimeUnixNano,
+			SpanID:       spanItem.SpanID,
+			TraceID:      spanItem.TraceID,
+			ServiceName:  spanItem.ServiceName,
+			Name:         spanItem.Name,
+			Kind:         spanItem.Kind,
+			DurationNano: spanItem.DurationNano,
+			TagMap:       spanItem.TagMap,
+			ParentID:     parentID,
+			Events:       spanItem.Events,
+			HasError:     spanItem.HasError,
+		}
+		spans = append(spans, span)
+	}
+
+	// Build span trees from the spans
+	roots, err := buildSpanTrees(&spans)
+	if err != nil {
+		return nil, err
+	}
+	targetSpan := &model.SpanForTraceDetails{}
+
+	// Find the target span in the span trees
+	for _, root := range roots {
+		targetSpan, err = breadthFirstSearch(root, targetSpanId)
+		if targetSpan != nil {
+			break
+		}
+		if err != nil {
+			zap.S().Error("Error during BreadthFirstSearch(): ", err)
+			return nil, err
+		}
+	}
+
+	// If the target span is not found, return span not found error
+	if targetSpan == nil {
+		return nil, errors.New("Span not found")
+	}
+
+	// Build the final result
+	parents := []*model.SpanForTraceDetails{}
+
+	// Get the parent spans of the target span up to the given levelUp parameter and spanLimit
+	preParent := targetSpan
+	for i := 0; i < levelUp+1; i++ {
+		if i == levelUp {
+			preParent.ParentID = ""
+		}
+		if spanLimit-len(preParent.Children) <= 0 {
+			parents = append(parents, preParent)
+			parents = append(parents, preParent.Children[:spanLimit]...)
+			spanLimit -= (len(preParent.Children[:spanLimit]) + 1)
+			preParent.ParentID = ""
+			break
+		}
+		parents = append(parents, preParent)
+		parents = append(parents, preParent.Children...)
+		spanLimit -= (len(preParent.Children) + 1)
+		preParent = preParent.ParentSpan
+		if preParent == nil {
+			break
+		}
+	}
+
+	// Get the child spans of the target span until the given levelDown and spanLimit
+	preParents := []*model.SpanForTraceDetails{targetSpan}
+	children := []*model.SpanForTraceDetails{}
+
+	for i := 0; i < levelDown && len(preParents) != 0 && spanLimit > 0; i++ {
+		parents := []*model.SpanForTraceDetails{}
+		for _, parent := range preParents {
+			if spanLimit-len(parent.Children) <= 0 {
+				children = append(children, parent.Children[:spanLimit]...)
+				spanLimit -= len(parent.Children[:spanLimit])
+				break
+			}
+			children = append(children, parent.Children...)
+			parents = append(parents, parent.Children...)
+		}
+		preParents = parents
+	}
+
+	// Store the final list of spans in the resultSpanSet map to avoid duplicates
+	resultSpansSet := make(map[*model.SpanForTraceDetails]struct{})
+	resultSpansSet[targetSpan] = struct{}{}
+	for _, parent := range parents {
+		resultSpansSet[parent] = struct{}{}
+	}
+	for _, child := range children {
+		resultSpansSet[child] = struct{}{}
+	}
+
+	searchSpansResult := []basemodel.SearchSpansResult{{
+		Columns: []string{"__time", "SpanId", "TraceId", "ServiceName", "Name", "Kind", "DurationNano", "TagsKeys", "TagsValues", "References", "Events", "HasError"},
+		Events:  make([][]interface{}, len(resultSpansSet)),
+	},
+	}
+
+	// Convert the resultSpansSet map to searchSpansResult
+	i := 0 // index for spans
+	for item := range resultSpansSet {
+		references := []basemodel.OtelSpanRef{
+			{
+				TraceId: item.TraceID,
+				SpanId:  item.ParentID,
+				RefType: "CHILD_OF",
+			},
+		}
+
+		referencesStringArray := []string{}
+		for _, item := range references {
+			referencesStringArray = append(referencesStringArray, item.ToString())
+		}
+		keys := make([]string, 0, len(item.TagMap))
+		values := make([]string, 0, len(item.TagMap))
+
+		for k, v := range item.TagMap {
+			keys = append(keys, k)
+			values = append(values, v)
+		}
+		if item.Events == nil {
+			item.Events = []string{}
+		}
+		searchSpansResult[0].Events[i] = []interface{}{
+			item.TimeUnixNano,
+			item.SpanID,
+			item.TraceID,
+			item.ServiceName,
+			item.Name,
+			strconv.Itoa(int(item.Kind)),
+			strconv.FormatInt(item.DurationNano, 10),
+			keys,
+			values,
+			referencesStringArray,
+			item.Events,
+			item.HasError,
+		}
+		i++ // increment index
+	}
+	return searchSpansResult, nil
+}
+
+// buildSpanTrees builds trees of spans from a list of spans.
+func buildSpanTrees(spansPtr *[]*model.SpanForTraceDetails) ([]*model.SpanForTraceDetails, error) {
+
+	// Build a map of spanID to span for fast lookup
+	var roots []*model.SpanForTraceDetails
+	spans := *spansPtr
+	mapOfSpans := make(map[string]*model.SpanForTraceDetails, len(spans))
+
+	for _, span := range spans {
+		if span.ParentID == "" {
+			roots = append(roots, span)
+		}
+		mapOfSpans[span.SpanID] = span
+	}
+
+	// Build the span tree by adding children to the parent spans
+	for _, span := range spans {
+		if span.ParentID == "" {
+			continue
+		}
+		parent := mapOfSpans[span.ParentID]
+
+		// If the parent span is not found, add current span to list of roots
+		if parent == nil {
+			// zap.S().Debug("Parent Span not found parent_id: ", span.ParentID)
+			roots = append(roots, span)
+			span.ParentID = ""
+			continue
+		}
+
+		span.ParentSpan = parent
+		parent.Children = append(parent.Children, span)
+	}
+
+	return roots, nil
+}
+
+// breadthFirstSearch performs a breadth-first search on the span tree to find the target span.
+func breadthFirstSearch(spansPtr *model.SpanForTraceDetails, targetId string) (*model.SpanForTraceDetails, error) {
+	queue := []*model.SpanForTraceDetails{spansPtr}
+	visited := make(map[string]bool)
+
+	for len(queue) > 0 {
+		current := queue[0]
+		visited[current.SpanID] = true
+		queue = queue[1:]
+		if current.SpanID == targetId {
+			return current, nil
+		}
+
+		for _, child := range current.Children {
+			if ok, _ := visited[child.SpanID]; !ok {
+				queue = append(queue, child)
+			}
+		}
+	}
+	return nil, nil
+}

ee/query-service/app/server.go

@@ -99,7 +99,7 @@ func NewServer(serverOptions *ServerOptions) (*Server, error) {
 	storage := os.Getenv("STORAGE")
 	if storage == "clickhouse" {
 		zap.S().Info("Using ClickHouse as datastore ...")
-		qb := db.NewDataConnector(localDB, serverOptions.PromConfigPath)
+		qb := db.NewDataConnector(localDB, serverOptions.PromConfigPath, lm)
 		go qb.Start(readerReady)
 		reader = qb
 	} else {

ee/query-service/constants/constants.go

@@ -10,6 +10,8 @@ const (
 
 var LicenseSignozIo = "https://license.signoz.io/api/v1"
 
+var SpanLimitStr = GetOrDefaultEnv("SPAN_LIMIT", "5000")
+
 func GetOrDefaultEnv(key string, fallback string) string {
 	v := os.Getenv(key)
 	if len(v) == 0 {

ee/query-service/dao/interface.go

@@ -2,7 +2,7 @@ package dao
 
 import (
 	"context"
-
+	"net/url"
 	"github.com/google/uuid"
 	"github.com/jmoiron/sqlx"
 	"go.signoz.io/signoz/ee/query-service/model"
@@ -22,7 +22,9 @@ type ModelDao interface {
 	// auth methods
 	PrecheckLogin(ctx context.Context, email, sourceUrl string) (*model.PrecheckResponse, basemodel.BaseApiError)
 	CanUsePassword(ctx context.Context, email string) (bool, basemodel.BaseApiError)
-
+	PrepareSsoRedirect(ctx context.Context, redirectUri, email string) (redirectURL string, apierr basemodel.BaseApiError)
+	GetDomainFromSsoResponse(ctx context.Context, relayState *url.URL) (*model.OrgDomain, error)
+	
 	// org domain (auth domains) CRUD ops
 	ListDomains(ctx context.Context, orgId string) ([]model.OrgDomain, basemodel.BaseApiError)
 	GetDomain(ctx context.Context, id uuid.UUID) (*model.OrgDomain, basemodel.BaseApiError)

ee/query-service/dao/sqlite/auth.go

@@ -10,9 +10,33 @@ import (
 	"go.signoz.io/signoz/ee/query-service/model"
 	baseconst "go.signoz.io/signoz/pkg/query-service/constants"
 	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	baseauth "go.signoz.io/signoz/pkg/query-service/auth"
 	"go.uber.org/zap"
 )
 
+// PrepareSsoRedirect prepares redirect page link after SSO response 
+// is successfully parsed (i.e. valid email is available)
+func (m *modelDao) PrepareSsoRedirect(ctx context.Context, redirectUri, email string) (redirectURL string, apierr basemodel.BaseApiError) {
+
+	userPayload, apierr := m.GetUserByEmail(ctx, email)
+	if !apierr.IsNil() {
+		zap.S().Errorf(" failed to get user with email received from auth provider", apierr.Error())
+		return "", model.BadRequestStr("invalid user email received from the auth provider")
+	}
+
+	tokenStore, err := baseauth.GenerateJWTForUser(&userPayload.User)
+	if err != nil {
+		zap.S().Errorf("failed to generate token for SSO login user", err)
+		return "", model.InternalErrorStr("failed to generate token for the user")
+	}
+
+	return fmt.Sprintf("%s?jwt=%s&usr=%s&refreshjwt=%s",
+		redirectUri,
+		tokenStore.AccessJwt,
+		userPayload.User.Id,
+		tokenStore.RefreshJwt), nil
+}
+
 func (m *modelDao) CanUsePassword(ctx context.Context, email string) (bool, basemodel.BaseApiError) {
 	domain, apierr := m.GetDomainByEmail(ctx, email)
 	if apierr != nil {

ee/query-service/dao/sqlite/domain.go

@@ -4,6 +4,7 @@ import (
 	"context"
 	"database/sql"
 	"encoding/json"
+	"net/url"
 	"fmt"
 	"strings"
 	"time"
@@ -25,6 +26,34 @@ type StoredDomain struct {
 	UpdatedAt int64     `db:"updated_at"`
 }
 
+// GetDomainFromSsoResponse uses relay state received from IdP to fetch
+// user domain. The domain is further used to process validity of the response.
+// when sending login request to IdP we send relay state as URL (site url) 
+// with domainId as query parameter.   
+func (m *modelDao) GetDomainFromSsoResponse(ctx context.Context, relayState *url.URL) (*model.OrgDomain, error) {
+	// derive domain id from relay state now
+	var domainIdStr string 
+	for k, v := range relayState.Query() {
+		if k == "domainId" && len(v) > 0 {
+			domainIdStr = strings.Replace(v[0], ":", "-", -1)
+		}
+	}
+
+	domainId, err := uuid.Parse(domainIdStr)
+	if err != nil {
+		zap.S().Errorf("failed to parse domain id from relay state", err)
+		return nil, fmt.Errorf("failed to parse response from IdP response")
+	}
+
+	domain, err := m.GetDomain(ctx, domainId)
+	if (err != nil) || domain == nil {
+		zap.S().Errorf("failed to find domain received in IdP response", err.Error())
+		return nil, fmt.Errorf("invalid credentials")
+	}
+
+	return domain, nil
+}
+
 // GetDomain returns org domain for a given domain id
 func (m *modelDao) GetDomain(ctx context.Context, id uuid.UUID) (*model.OrgDomain, basemodel.BaseApiError) {
 

ee/query-service/integrations/signozio/signozio.go

@@ -127,7 +127,7 @@ func NewPostRequestWithCtx(ctx context.Context, url string, contentType string,
 }
 
 // SendUsage reports the usage of signoz to license server
-func SendUsage(ctx context.Context, usage *model.UsagePayload) *model.ApiError {
+func SendUsage(ctx context.Context, usage model.UsagePayload) *model.ApiError {
 	reqString, _ := json.Marshal(usage)
 	req, err := NewPostRequestWithCtx(ctx, C.Prefix+"/usage", APPLICATION_JSON, bytes.NewBuffer(reqString))
 	if err != nil {

ee/query-service/license/manager.go

@@ -10,6 +10,8 @@ import (
 
 	"sync"
 
+	baseconstants "go.signoz.io/signoz/pkg/query-service/constants"
+
 	validate "go.signoz.io/signoz/ee/query-service/integrations/signozio"
 	"go.signoz.io/signoz/ee/query-service/model"
 	basemodel "go.signoz.io/signoz/pkg/query-service/model"
@@ -92,6 +94,8 @@ func (lm *Manager) SetActive(l *model.License) {
 
 	lm.activeLicense = l
 	lm.activeFeatures = l.FeatureSet
+	// set default features
+	setDefaultFeatures(lm)
 	if !lm.validatorRunning {
 		// we want to make sure only one validator runs,
 		// we already have lock() so good to go
@@ -101,7 +105,13 @@ func (lm *Manager) SetActive(l *model.License) {
 
 }
 
-// LoadActiveLicense loads the most recent active licenseex
+func setDefaultFeatures(lm *Manager) {
+	for k, v := range baseconstants.DEFAULT_FEATURE_SET {
+		lm.activeFeatures[k] = v
+	}
+}
+
+// LoadActiveLicense loads the most recent active license
 func (lm *Manager) LoadActiveLicense() error {
 	var err error
 	active, err := lm.repo.GetActiveLicense(context.Background())
@@ -111,7 +121,10 @@ func (lm *Manager) LoadActiveLicense() error {
 	if active != nil {
 		lm.SetActive(active)
 	} else {
-		zap.S().Info("No active license found.")
+		zap.S().Info("No active license found, defaulting to basic plan")
+		// if no active license is found, we default to basic(free) plan with all default features
+		lm.activeFeatures = basemodel.BasicPlan
+		setDefaultFeatures(lm)
 	}
 
 	return nil
@@ -278,8 +291,11 @@ func (lm *Manager) Activate(ctx context.Context, key string) (licenseResponse *m
 // CheckFeature will be internally used by backend routines
 // for feature gating
 func (lm *Manager) CheckFeature(featureKey string) error {
-	if _, ok := lm.activeFeatures[featureKey]; ok {
-		return nil
+	if value, ok := lm.activeFeatures[featureKey]; ok {
+		if value {
+			return nil
+		}
+		return basemodel.ErrFeatureUnavailable{Key: featureKey}
 	}
 	return basemodel.ErrFeatureUnavailable{Key: featureKey}
 }

ee/query-service/model/domain.go

@@ -9,8 +9,10 @@ import (
 	"github.com/google/uuid"
 	"github.com/pkg/errors"
 	saml2 "github.com/russellhaering/gosaml2"
-	"go.signoz.io/signoz/ee/query-service/saml"
+	"go.signoz.io/signoz/ee/query-service/sso/saml"
+	"go.signoz.io/signoz/ee/query-service/sso"
 	basemodel "go.signoz.io/signoz/pkg/query-service/model"
+	"go.uber.org/zap"
 )
 
 type SSOType string
@@ -20,12 +22,6 @@ const (
 	GoogleAuth SSOType = "GOOGLE_AUTH"
 )
 
-type SamlConfig struct {
-	SamlEntity string `json:"samlEntity"`
-	SamlIdp    string `json:"samlIdp"`
-	SamlCert   string `json:"samlCert"`
-}
-
 // OrgDomain identify org owned web domains for auth and other purposes
 type OrgDomain struct {
 	Id         uuid.UUID   `json:"id"`
@@ -33,10 +29,17 @@ type OrgDomain struct {
 	OrgId      string      `json:"orgId"`
 	SsoEnabled bool        `json:"ssoEnabled"`
 	SsoType    SSOType     `json:"ssoType"`
+
 	SamlConfig *SamlConfig `json:"samlConfig"`
+	GoogleAuthConfig *GoogleOAuthConfig `json:"googleAuthConfig"`
+
 	Org        *basemodel.Organization
 }
 
+func (od *OrgDomain) String() string {
+	return fmt.Sprintf("[%s]%s-%s ", od.Name, od.Id.String(), od.SsoType)
+}
+
 // Valid is used a pipeline function to check if org domain
 // loaded from db is valid
 func (od *OrgDomain) Valid(err error) error {
@@ -97,6 +100,16 @@ func (od *OrgDomain) GetSAMLCert() string {
 	return ""
 }
 
+// PrepareGoogleOAuthProvider creates GoogleProvider that is used in 
+// requesting OAuth and also used in processing response from google 
+func (od *OrgDomain) PrepareGoogleOAuthProvider(siteUrl *url.URL) (sso.OAuthCallbackProvider, error) {
+	if od.GoogleAuthConfig == nil {
+		return nil, fmt.Errorf("Google auth is not setup correctly for this domain")
+	}
+
+	return od.GoogleAuthConfig.GetProvider(od.Name, siteUrl)
+}
+
 // PrepareSamlRequest creates a request accordingly gosaml2
 func (od *OrgDomain) PrepareSamlRequest(siteUrl *url.URL) (*saml2.SAMLServiceProvider, error) {
 
@@ -124,19 +137,48 @@ func (od *OrgDomain) PrepareSamlRequest(siteUrl *url.URL) (*saml2.SAMLServicePro
 }
 
 func (od *OrgDomain) BuildSsoUrl(siteUrl *url.URL) (ssoUrl string, err error) {
-
-	sp, err := od.PrepareSamlRequest(siteUrl)
-	if err != nil {
-		return "", err
-	}
+	
 
 	fmtDomainId := strings.Replace(od.Id.String(), "-", ":", -1)
+	
+	// build redirect url from window.location sent by frontend
+	redirectURL := fmt.Sprintf("%s://%s%s", siteUrl.Scheme, siteUrl.Host, siteUrl.Path)
+
+	// prepare state that gets relayed back when the auth provider
+	// calls back our url. here we pass the app url (where signoz runs)
+	// and the domain Id. The domain Id helps in identifying sso config
+	// when the call back occurs and the app url is useful in redirecting user 
+	// back to the right path. 
+	// why do we need to pass app url? the callback typically is handled by backend
+	// and sometimes backend might right at a different port or is unaware of frontend
+	// endpoint (unless SITE_URL param is set). hence, we receive this build sso request
+	// along with frontend window.location and use it to relay the information through 
+	// auth provider to the backend (HandleCallback or HandleSSO method). 
+	relayState := fmt.Sprintf("%s?domainId=%s", redirectURL, fmtDomainId)
+		
+
+	switch (od.SsoType) {
+	case SAML:
+
+		sp, err := od.PrepareSamlRequest(siteUrl)
+		if err != nil {
+			return "", err
+		}
+	
+		return sp.BuildAuthURL(relayState)
+	
+	case GoogleAuth:
+		
+		googleProvider, err := od.PrepareGoogleOAuthProvider(siteUrl)
+		if err != nil {
+			return "", err
+		}
+		return googleProvider.BuildAuthURL(relayState)
+
+	default:
+		zap.S().Errorf("found unsupported SSO config for the org domain", zap.String("orgDomain", od.Name))
+		return "", fmt.Errorf("unsupported SSO config for the domain") 
+	}
 
-	relayState := fmt.Sprintf("%s://%s%s?domainId=%s",
-		siteUrl.Scheme,
-		siteUrl.Host,
-		siteUrl.Path,
-		fmtDomainId)
 
-	return sp.BuildAuthURL(relayState)
 }

ee/query-service/model/errors.go

@@ -1,6 +1,7 @@
 package model
 
 import (
+	"fmt"
 	basemodel "go.signoz.io/signoz/pkg/query-service/model"
 )
 
@@ -44,6 +45,14 @@ func BadRequest(err error) *ApiError {
 	}
 }
 
+// BadRequestStr returns a ApiError object of bad request for string input
+func BadRequestStr(s string) *ApiError {
+	return &ApiError{
+		Typ: basemodel.ErrorBadData,
+		Err: fmt.Errorf(s),
+	}
+}
+
 // InternalError returns a ApiError object of internal type
 func InternalError(err error) *ApiError {
 	return &ApiError{
@@ -52,6 +61,14 @@ func InternalError(err error) *ApiError {
 	}
 }
 
+
+// InternalErrorStr returns a ApiError object of internal type for string input
+func InternalErrorStr(s string) *ApiError {
+	return &ApiError{
+		Typ: basemodel.ErrorInternal,
+		Err: fmt.Errorf(s),
+	}
+}
 var (
 	ErrorNone                  basemodel.ErrorType = ""
 	ErrorTimeout               basemodel.ErrorType = "timeout"

ee/query-service/model/plans.go

@@ -17,11 +17,15 @@ var BasicPlan = basemodel.FeatureSet{
 }
 
 var ProPlan = basemodel.FeatureSet{
-	Pro: true,
-	SSO: true,
+	Pro:                             true,
+	SSO:                             true,
+	basemodel.SmartTraceDetail:      true,
+	basemodel.CustomMetricsFunction: true,
 }
 
 var EnterprisePlan = basemodel.FeatureSet{
-	Enterprise: true,
-	SSO:        true,
+	Enterprise:                      true,
+	SSO:                             true,
+	basemodel.SmartTraceDetail:      true,
+	basemodel.CustomMetricsFunction: true,
 }

ee/query-service/model/sso.go

@@ -0,0 +1,68 @@
+package model
+
+import (
+	"fmt"
+	"context"
+	"net/url"
+	"golang.org/x/oauth2"
+	"github.com/coreos/go-oidc/v3/oidc"
+	"go.signoz.io/signoz/ee/query-service/sso"
+)
+
+// SamlConfig contans SAML params to generate and respond to the requests 
+// from SAML provider
+type SamlConfig struct {
+	SamlEntity string `json:"samlEntity"`
+	SamlIdp    string `json:"samlIdp"`
+	SamlCert   string `json:"samlCert"`
+}
+
+// GoogleOauthConfig contains a generic config to support oauth 
+type GoogleOAuthConfig struct {
+	ClientID     string `json:"clientId"`
+	ClientSecret string `json:"clientSecret"`
+	RedirectURI  string `json:"redirectURI"`
+}
+
+
+const (
+	googleIssuerURL = "https://accounts.google.com"
+)
+
+func (g *GoogleOAuthConfig) GetProvider(domain string, siteUrl *url.URL) (sso.OAuthCallbackProvider, error) {
+
+	ctx, cancel := context.WithCancel(context.Background())
+
+	provider, err := oidc.NewProvider(ctx, googleIssuerURL)
+	if err != nil {
+		cancel()
+		return nil, fmt.Errorf("failed to get provider: %v", err)
+	}
+
+	// default to email and profile scope as we just use google auth
+	// to verify identity and start a session. 
+	scopes := []string{"email"}
+
+	// this is the url google will call after login completion
+	redirectURL := fmt.Sprintf("%s://%s/%s",
+		siteUrl.Scheme,
+		siteUrl.Host,
+		"api/v1/complete/google")
+
+	return &sso.GoogleOAuthProvider{
+		RedirectURI: g.RedirectURI,
+		OAuth2Config: &oauth2.Config{
+			ClientID:     g.ClientID,
+			ClientSecret: g.ClientSecret,
+			Endpoint:     provider.Endpoint(),
+			Scopes:       scopes,
+			RedirectURL:  redirectURL,
+		},
+		Verifier: provider.Verifier(
+			&oidc.Config{ClientID: g.ClientID},
+		),
+		Cancel:                         cancel,
+		HostedDomain:                  domain,
+	}, nil
+}
+

ee/query-service/model/trace.go

@@ -0,0 +1,22 @@
+package model
+
+type SpanForTraceDetails struct {
+	TimeUnixNano uint64                 `json:"timestamp"`
+	SpanID       string                 `json:"spanID"`
+	TraceID      string                 `json:"traceID"`
+	ParentID     string                 `json:"parentID"`
+	ParentSpan   *SpanForTraceDetails   `json:"parentSpan"`
+	ServiceName  string                 `json:"serviceName"`
+	Name         string                 `json:"name"`
+	Kind         int32                  `json:"kind"`
+	DurationNano int64                  `json:"durationNano"`
+	TagMap       map[string]string      `json:"tagMap"`
+	Events       []string               `json:"event"`
+	HasError     bool                   `json:"hasError"`
+	Children     []*SpanForTraceDetails `json:"children"`
+}
+
+type GetSpansSubQueryDBResponse struct {
+	SpanID  string `ch:"spanID"`
+	TraceID string `ch:"traceID"`
+}

ee/query-service/model/usage.go

@@ -6,30 +6,27 @@ import (
 	"github.com/google/uuid"
 )
 
-type UsageSnapshot struct {
-	CurrentLogSizeBytes            uint64 `json:"currentLogSizeBytes"`
-	CurrentLogSizeBytesColdStorage uint64 `json:"currentLogSizeBytesColdStorage"`
-	CurrentSpansCount              uint64 `json:"currentSpansCount"`
-	CurrentSpansCountColdStorage   uint64 `json:"currentSpansCountColdStorage"`
-	CurrentSamplesCount            uint64 `json:"currentSamplesCount"`
-	CurrentSamplesCountColdStorage uint64 `json:"currentSamplesCountColdStorage"`
-}
-
-type UsageBase struct {
-	Id                uuid.UUID `json:"id" db:"id"`
-	InstallationId    uuid.UUID `json:"installationId" db:"installation_id"`
-	ActivationId      uuid.UUID `json:"activationId" db:"activation_id"`
-	CreatedAt         time.Time `json:"createdAt" db:"created_at"`
-	FailedSyncRequest int       `json:"failedSyncRequest" db:"failed_sync_request_count"`
-}
-
 type UsagePayload struct {
-	UsageBase
-	Metrics      UsageSnapshot `json:"metrics"`
-	SnapshotDate time.Time     `json:"snapshotDate"`
+	InstallationId uuid.UUID `json:"installationId"`
+	LicenseKey     uuid.UUID `json:"licenseKey"`
+	Usage          []Usage   `json:"usage"`
 }
 
 type Usage struct {
-	UsageBase
-	Snapshot string `db:"snapshot"`
+	CollectorID string    `json:"collectorId"`
+	ExporterID  string    `json:"exporterId"`
+	Type        string    `json:"type"`
+	Tenant      string    `json:"tenant"`
+	TimeStamp   time.Time `json:"timestamp"`
+	Count       int64     `json:"count"`
+	Size        int64     `json:"size"`
+}
+
+type UsageDB struct {
+	CollectorID string    `ch:"collector_id" json:"collectorId"`
+	ExporterID  string    `ch:"exporter_id" json:"exporterId"`
+	Type        string    `ch:"-" json:"type"`
+	TimeStamp   time.Time `ch:"timestamp" json:"timestamp"`
+	Tenant      string    `ch:"tenant" json:"tenant"`
+	Data        string    `ch:"data" json:"data"`
 }

ee/query-service/sso/google.go

@@ -0,0 +1,92 @@
+package sso
+
+import (
+	"fmt"
+	"errors"
+	"context"
+	"net/http"
+	"github.com/coreos/go-oidc/v3/oidc"
+	"golang.org/x/oauth2"
+)
+
+type GoogleOAuthProvider struct {
+	RedirectURI                    string
+	OAuth2Config                   *oauth2.Config
+	Verifier                       *oidc.IDTokenVerifier
+	Cancel                         context.CancelFunc
+	HostedDomain 				   string
+}
+
+
+func (g *GoogleOAuthProvider) BuildAuthURL(state string) (string, error) {
+	var opts []oauth2.AuthCodeOption
+	
+	// set hosted domain. google supports multiple hosted domains but in our case
+	// we have one config per host domain. 
+	opts = append(opts, oauth2.SetAuthURLParam("hd", g.HostedDomain))
+
+	return g.OAuth2Config.AuthCodeURL(state, opts...), nil
+}
+
+type oauth2Error struct{
+	error            string
+	errorDescription string
+}
+
+func (e *oauth2Error) Error() string {
+	if e.errorDescription == "" {
+		return e.error
+	}
+	return e.error + ": " + e.errorDescription
+}
+
+func (g *GoogleOAuthProvider) HandleCallback(r *http.Request) (identity *SSOIdentity, err error) {
+	q := r.URL.Query()
+	if errType := q.Get("error"); errType != "" {
+		return identity, &oauth2Error{errType, q.Get("error_description")}
+	}
+
+	token, err := g.OAuth2Config.Exchange(r.Context(), q.Get("code"))
+	if err != nil {
+		return identity, fmt.Errorf("google: failed to get token: %v", err)
+	}
+
+	return g.createIdentity(r.Context(), token)
+}
+
+	
+func (g *GoogleOAuthProvider) createIdentity(ctx context.Context, token *oauth2.Token) (identity *SSOIdentity, err error) {
+	rawIDToken, ok := token.Extra("id_token").(string)
+	if !ok {
+		return identity, errors.New("google: no id_token in token response")
+	}
+	idToken, err := g.Verifier.Verify(ctx, rawIDToken)
+	if err != nil {
+		return identity, fmt.Errorf("google: failed to verify ID Token: %v", err)
+	}
+
+	var claims struct {
+		Username      string `json:"name"`
+		Email         string `json:"email"`
+		EmailVerified bool   `json:"email_verified"`
+		HostedDomain  string `json:"hd"`
+	}
+	if err := idToken.Claims(&claims); err != nil {
+		return identity, fmt.Errorf("oidc: failed to decode claims: %v", err)
+	}
+
+	if claims.HostedDomain != g.HostedDomain {
+		return identity, fmt.Errorf("oidc: unexpected hd claim %v", claims.HostedDomain)	
+	}
+
+	identity = &SSOIdentity{
+		UserID:        idToken.Subject,
+		Username:      claims.Username,
+		Email:         claims.Email,
+		EmailVerified: claims.EmailVerified,
+		ConnectorData: []byte(token.RefreshToken),
+	}
+
+	return identity, nil
+}
+

ee/query-service/sso/model.go

@@ -0,0 +1,31 @@
+package sso 
+
+import (
+	"net/http"
+)
+
+// SSOIdentity contains details of user received from SSO provider 
+type SSOIdentity struct {
+	UserID            string
+	Username          string
+	PreferredUsername string
+	Email             string
+	EmailVerified     bool
+	ConnectorData []byte
+}
+
+// OAuthCallbackProvider is an interface implemented by connectors which use an OAuth
+// style redirect flow to determine user information.
+type OAuthCallbackProvider interface {
+	// The initial URL user would be redirect to.  
+	// OAuth2 implementations support various scopes but we only need profile and user as 
+	// the roles are still being managed in SigNoz. 
+	BuildAuthURL(state string) (string, error)
+
+	// Handle the callback to the server (after login at oauth provider site)
+	// and return a email identity. 
+	// At the moment we dont support auto signup flow (based on domain), so 
+	// the full identity (including name, group etc) is not required outside of the 
+	// connector 
+	HandleCallback(r *http.Request) (identity *SSOIdentity, err error)
+}

ee/query-service/usage/manager.go

@@ -4,18 +4,19 @@ import (
 	"context"
 	"encoding/json"
 	"fmt"
+	"strings"
 	"sync/atomic"
 	"time"
 
 	"github.com/ClickHouse/clickhouse-go/v2"
 	"github.com/google/uuid"
 	"github.com/jmoiron/sqlx"
+
 	"go.uber.org/zap"
 
 	licenseserver "go.signoz.io/signoz/ee/query-service/integrations/signozio"
 	"go.signoz.io/signoz/ee/query-service/license"
 	"go.signoz.io/signoz/ee/query-service/model"
-	"go.signoz.io/signoz/ee/query-service/usage/repository"
 	"go.signoz.io/signoz/pkg/query-service/utils/encryption"
 )
 
@@ -27,9 +28,6 @@ const (
 )
 
 var (
-	// collect usage every hour
-	collectionFrequency = 1 * time.Hour
-
 	// send usage every 24 hour
 	uploadFrequency = 24 * time.Hour
 
@@ -37,8 +35,6 @@ var (
 )
 
 type Manager struct {
-	repository *repository.Repository
-
 	clickhouseConn clickhouse.Conn
 
 	licenseRepo *license.Repo
@@ -52,15 +48,9 @@ type Manager struct {
 }
 
 func New(dbType string, db *sqlx.DB, licenseRepo *license.Repo, clickhouseConn clickhouse.Conn) (*Manager, error) {
-	repo := repository.New(db)
-
-	err := repo.Init(dbType)
-	if err != nil {
-		return nil, fmt.Errorf("failed to initiate usage repo: %v", err)
-	}
 
 	m := &Manager{
-		repository:     repo,
+		// repository:     repo,
 		clickhouseConn: clickhouseConn,
 		licenseRepo:    licenseRepo,
 	}
@@ -74,6 +64,28 @@ func (lm *Manager) Start() error {
 		return fmt.Errorf("usage exporter is locked")
 	}
 
+	go lm.UsageExporter(context.Background())
+
+	return nil
+}
+
+func (lm *Manager) UsageExporter(ctx context.Context) {
+	defer close(lm.terminated)
+
+	uploadTicker := time.NewTicker(uploadFrequency)
+	defer uploadTicker.Stop()
+
+	for {
+		select {
+		case <-lm.done:
+			return
+		case <-uploadTicker.C:
+			lm.UploadUsage(ctx)
+		}
+	}
+}
+
+func (lm *Manager) UploadUsage(ctx context.Context) error {
 	// check if license is present or not
 	license, err := lm.licenseRepo.GetActiveLicense(context.Background())
 	if err != nil {
@@ -85,203 +97,81 @@ func (lm *Manager) Start() error {
 		return nil
 	}
 
-	// upload previous snapshots if any
-	err = lm.UploadUsage(context.Background())
-	if err != nil {
-		return err
-	}
-
-	// collect snapshot if incase it wasn't collect in (t - collectionFrequency)
-	err = lm.CollectCurrentUsage(context.Background())
-	if err != nil {
-		return err
-	}
-
-	go lm.UsageExporter(context.Background())
-
-	return nil
-}
-
-// CollectCurrentUsage checks if needs to collect usage data
-func (lm *Manager) CollectCurrentUsage(ctx context.Context) error {
-	// check the DB if anything exist  where timestamp > t - collectionFrequency
-	ts := time.Now().Add(-collectionFrequency)
-	alreadyCreated, err := lm.repository.CheckSnapshotGtCreatedAt(ctx, ts)
-	if err != nil {
-		return err
-	}
-	if !alreadyCreated {
-		zap.S().Info("Collecting current usage")
-		exportError := lm.CollectAndStoreUsage(ctx)
-		if exportError != nil {
-			return exportError
-		}
-	} else {
-		zap.S().Info("Nothing to collect")
-	}
-	return nil
-}
-
-func (lm *Manager) UsageExporter(ctx context.Context) {
-	defer close(lm.terminated)
-
-	collectionTicker := time.NewTicker(collectionFrequency)
-	defer collectionTicker.Stop()
-
-	uploadTicker := time.NewTicker(uploadFrequency)
-	defer uploadTicker.Stop()
-
-	for {
-		select {
-		case <-lm.done:
-			return
-		case <-collectionTicker.C:
-			lm.CollectAndStoreUsage(ctx)
-		case <-uploadTicker.C:
-			lm.UploadUsage(ctx)
-			// remove the old snapshots
-			lm.repository.DropOldSnapshots(ctx)
-		}
-	}
-}
-
-type TableSize struct {
-	Table             string `ch:"table"`
-	DiskName          string `ch:"disk_name"`
-	Rows              uint64 `ch:"rows"`
-	UncompressedBytes uint64 `ch:"uncompressed_bytes"`
-}
-
-func (lm *Manager) CollectAndStoreUsage(ctx context.Context) error {
-	snap, err := lm.GetUsageFromClickHouse(ctx)
-	if err != nil {
-		return err
-	}
-
-	license, err := lm.licenseRepo.GetActiveLicense(ctx)
-	if err != nil {
-		return err
-	}
-
-	activationId, _ := uuid.Parse(license.ActivationId)
-	// TODO (nitya) : Add installation ID in the payload
-	payload := model.UsagePayload{
-		UsageBase: model.UsageBase{
-			ActivationId:      activationId,
-			FailedSyncRequest: 0,
-		},
-		Metrics:      *snap,
-		SnapshotDate: time.Now(),
-	}
-
-	err = lm.repository.InsertSnapshot(ctx, &payload)
-	if err != nil {
-		return err
-	}
-
-	return nil
-}
-
-func (lm *Manager) GetUsageFromClickHouse(ctx context.Context) (*model.UsageSnapshot, error) {
-	tableSizes := []TableSize{}
-	snap := model.UsageSnapshot{}
+	usages := []model.UsageDB{}
 
 	// get usage from clickhouse
+	dbs := []string{"signoz_logs", "signoz_traces", "signoz_metrics"}
 	query := `
-		SELECT
-			table,
-			disk_name,
-			sum(rows) as rows,
-			sum(data_uncompressed_bytes) AS uncompressed_bytes
-		FROM system.parts
-		WHERE active AND (database in ('signoz_logs', 'signoz_metrics', 'signoz_traces')) AND (table in ('logs','samples_v2', 'signoz_index_v2'))
-		GROUP BY
-			table,
-			disk_name
-		ORDER BY table
+		SELECT tenant, collector_id, exporter_id, timestamp, data
+		FROM %s.distributed_usage as u1 
+			GLOBAL INNER JOIN 
+				(SELECT  
+					tenant, collector_id, exporter_id, MAX(timestamp) as ts 
+					FROM %s.distributed_usage as u2 
+					where timestamp >= $1 
+					GROUP BY tenant, collector_id, exporter_id 
+				) as t1
+		ON 
+		u1.tenant = t1.tenant AND u1.collector_id = t1.collector_id AND u1.exporter_id = t1.exporter_id and u1.timestamp = t1.ts 
+		order by timestamp
 	`
-	err := lm.clickhouseConn.Select(ctx, &tableSizes, query)
-	if err != nil {
-		return nil, err
-	}
 
-	for _, val := range tableSizes {
-		switch val.Table {
-		case "logs":
-			if val.DiskName == "default" {
-				snap.CurrentLogSizeBytes = val.UncompressedBytes
-			} else {
-				snap.CurrentLogSizeBytesColdStorage = val.UncompressedBytes
-			}
-		case "samples_v2":
-			if val.DiskName == "default" {
-				snap.CurrentSamplesCount = val.Rows
-			} else {
-				snap.CurrentSamplesCountColdStorage = val.Rows
-			}
-		case "signoz_index_v2":
-			if val.DiskName == "default" {
-				snap.CurrentSpansCount = val.Rows
-			} else {
-				snap.CurrentSpansCountColdStorage = val.Rows
-			}
+	for _, db := range dbs {
+		dbusages := []model.UsageDB{}
+		err := lm.clickhouseConn.Select(ctx, &dbusages, fmt.Sprintf(query, db, db), time.Now().Add(-(24 * time.Hour)))
+		if err != nil && !strings.Contains(err.Error(), "doesn't exist") {
+			return err
+		}
+		for _, u := range dbusages {
+			u.Type = db
+			usages = append(usages, u)
 		}
 	}
 
-	return &snap, nil
-}
-
-func (lm *Manager) UploadUsage(ctx context.Context) error {
-	snapshots, err := lm.repository.GetSnapshotsNotSynced(ctx)
-	if err != nil {
-		return err
-	}
-
-	if len(snapshots) <= 0 {
+	if len(usages) <= 0 {
 		zap.S().Info("no snapshots to upload, skipping.")
 		return nil
 	}
 
-	zap.S().Info("uploading snapshots")
-	for _, snap := range snapshots {
-		metricsBytes, err := encryption.Decrypt([]byte(snap.ActivationId.String()[:32]), []byte(snap.Snapshot))
+	zap.S().Info("uploading usage data")
+
+	usagesPayload := []model.Usage{}
+	for _, usage := range usages {
+		usageDataBytes, err := encryption.Decrypt([]byte(usage.ExporterID[:32]), []byte(usage.Data))
 		if err != nil {
 			return err
 		}
 
-		metrics := model.UsageSnapshot{}
-		err = json.Unmarshal(metricsBytes, &metrics)
+		usageData := model.Usage{}
+		err = json.Unmarshal(usageDataBytes, &usageData)
 		if err != nil {
 			return err
 		}
 
-		err = lm.UploadUsageWithExponentalBackOff(ctx, model.UsagePayload{
-			UsageBase: model.UsageBase{
-				Id:                snap.Id,
-				InstallationId:    snap.InstallationId,
-				ActivationId:      snap.ActivationId,
-				FailedSyncRequest: snap.FailedSyncRequest,
-			},
-			SnapshotDate: snap.CreatedAt,
-			Metrics:      metrics,
-		})
-		if err != nil {
-			return err
-		}
+		usageData.CollectorID = usage.CollectorID
+		usageData.ExporterID = usage.ExporterID
+		usageData.Type = usage.Type
+		usageData.Tenant = usage.Tenant
+		usagesPayload = append(usagesPayload, usageData)
+	}
+
+	key, _ := uuid.Parse(license.Key)
+	payload := model.UsagePayload{
+		LicenseKey: key,
+		Usage:      usagesPayload,
+	}
+	err = lm.UploadUsageWithExponentalBackOff(ctx, payload)
+	if err != nil {
+		return err
 	}
 	return nil
 }
 
 func (lm *Manager) UploadUsageWithExponentalBackOff(ctx context.Context, payload model.UsagePayload) error {
 	for i := 1; i <= MaxRetries; i++ {
-		apiErr := licenseserver.SendUsage(ctx, &payload)
+		apiErr := licenseserver.SendUsage(ctx, payload)
 		if apiErr != nil && i == MaxRetries {
-			err := lm.repository.IncrementFailedRequestCount(ctx, payload.Id)
-			if err != nil {
-				zap.S().Errorf("failed to updated the failure count for snapshot in DB : ", zap.Error(err))
-				return err
-			}
-			zap.S().Errorf("retries stopped : %v", zap.Error(err))
+			zap.S().Errorf("retries stopped : %v", zap.Error(apiErr))
 			// not returning error here since it is captured in the failed count
 			return nil
 		} else if apiErr != nil {
@@ -289,24 +179,10 @@ func (lm *Manager) UploadUsageWithExponentalBackOff(ctx context.Context, payload
 			sleepDuration := RetryInterval * time.Duration(i)
 			zap.S().Errorf("failed to upload snapshot retrying after %v secs : %v", sleepDuration.Seconds(), zap.Error(apiErr.Err))
 			time.Sleep(sleepDuration)
-
-			// update the failed request count
-			err := lm.repository.IncrementFailedRequestCount(ctx, payload.Id)
-			if err != nil {
-				zap.S().Errorf("failed to updated the failure count for snapshot in DB : %v", zap.Error(err))
-				return err
-			}
 		} else {
 			break
 		}
 	}
-
-	// update the database that it is synced
-	err := lm.repository.MoveToSynced(ctx, payload.Id)
-	if err != nil {
-		return err
-	}
-
 	return nil
 }
 

ee/query-service/usage/repository/repository.go

@@ -1,139 +0,0 @@
-package repository
-
-import (
-	"context"
-	"encoding/json"
-	"fmt"
-	"time"
-
-	"github.com/google/uuid"
-	"github.com/jmoiron/sqlx"
-	"go.uber.org/zap"
-
-	"go.signoz.io/signoz/ee/query-service/model"
-	"go.signoz.io/signoz/ee/query-service/usage/sqlite"
-	"go.signoz.io/signoz/pkg/query-service/utils/encryption"
-)
-
-const (
-	MaxFailedSyncCount = 9 // a snapshot will be ignored if the max failed count is greater than or equal to 9
-	SnapShotLife       = 3 * 24 * time.Hour
-)
-
-// Repository is usage Repository which stores usage snapshot in a secured DB
-type Repository struct {
-	db *sqlx.DB
-}
-
-// New initiates a new usage Repository
-func New(db *sqlx.DB) *Repository {
-	return &Repository{
-		db: db,
-	}
-}
-
-func (r *Repository) Init(engine string) error {
-	switch engine {
-	case "sqlite3", "sqlite":
-		return sqlite.InitDB(r.db)
-	default:
-		return fmt.Errorf("unsupported db")
-	}
-}
-
-func (r *Repository) InsertSnapshot(ctx context.Context, usage *model.UsagePayload) error {
-
-	snapshotBytes, err := json.Marshal(usage.Metrics)
-	if err != nil {
-		return err
-	}
-
-	usage.Id = uuid.New()
-
-	encryptedSnapshot, err := encryption.Encrypt([]byte(usage.ActivationId.String()[:32]), snapshotBytes)
-	if err != nil {
-		return err
-	}
-
-	query := `INSERT INTO usage(id, activation_id, snapshot)
- 				VALUES ($1, $2, $3)`
-	_, err = r.db.ExecContext(ctx,
-		query,
-		usage.Id,
-		usage.ActivationId,
-		string(encryptedSnapshot),
-	)
-	if err != nil {
-		zap.S().Errorf("error inserting usage data: %v", zap.Error(err))
-		return fmt.Errorf("failed to insert usage in db: %v", err)
-	}
-	return nil
-}
-
-func (r *Repository) MoveToSynced(ctx context.Context, id uuid.UUID) error {
-
-	query := `UPDATE usage 
-						SET synced = 'true',
-						synced_at = $1
-						WHERE id = $2`
-
-	_, err := r.db.ExecContext(ctx, query, time.Now(), id)
-
-	if err != nil {
-		zap.S().Errorf("error in updating usage: %v", zap.Error(err))
-		return fmt.Errorf("failed to update usage in db: %v", err)
-	}
-
-	return nil
-}
-
-func (r *Repository) IncrementFailedRequestCount(ctx context.Context, id uuid.UUID) error {
-
-	query := `UPDATE usage SET failed_sync_request_count = failed_sync_request_count + 1 WHERE id = $1`
-	_, err := r.db.ExecContext(ctx, query, id)
-	if err != nil {
-		zap.S().Errorf("error in updating usage: %v", zap.Error(err))
-		return fmt.Errorf("failed to update usage in db: %v", err)
-	}
-
-	return nil
-}
-
-func (r *Repository) GetSnapshotsNotSynced(ctx context.Context) ([]*model.Usage, error) {
-	snapshots := []*model.Usage{}
-
-	query := `SELECT id,created_at, activation_id, snapshot, failed_sync_request_count from usage where synced!='true' and failed_sync_request_count < $1 order by created_at asc `
-
-	err := r.db.SelectContext(ctx, &snapshots, query, MaxFailedSyncCount)
-	if err != nil {
-		return nil, err
-	}
-
-	return snapshots, nil
-}
-
-func (r *Repository) DropOldSnapshots(ctx context.Context) error {
-	query := `delete from usage where created_at <= $1`
-
-	_, err := r.db.ExecContext(ctx, query, time.Now().Add(-(SnapShotLife)))
-	if err != nil {
-		zap.S().Errorf("failed to remove old snapshots from db: %v", zap.Error(err))
-		return err
-	}
-
-	return nil
-}
-
-// CheckSnapshotGtCreatedAt checks if there is any snapshot greater than the provided timestamp
-func (r *Repository) CheckSnapshotGtCreatedAt(ctx context.Context, ts time.Time) (bool, error) {
-
-	var snapshots uint64
-	query := `SELECT count() from usage where created_at > '$1'`
-
-	err := r.db.QueryRowContext(ctx, query, ts).Scan(&snapshots)
-	if err != nil {
-		return false, err
-	}
-
-	return snapshots > 0, err
-}

ee/query-service/usage/sqlite/init.go

@@ -1,32 +0,0 @@
-package sqlite
-
-import (
-	"fmt"
-
-	"github.com/jmoiron/sqlx"
-)
-
-func InitDB(db *sqlx.DB) error {
-	var err error
-	if db == nil {
-		return fmt.Errorf("invalid db connection")
-	}
-
-	table_schema := `CREATE TABLE IF NOT EXISTS usage(
-		id UUID PRIMARY KEY,
-		created_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP, 
-		updated_at TIMESTAMP DEFAULT CURRENT_TIMESTAMP,
-		activation_id UUID, 
-		snapshot TEXT,
-		synced BOOLEAN DEFAULT 'false',
-		synced_at TIMESTAMP,
-		failed_sync_request_count INTEGER DEFAULT 0
-	);
-	`
-
-	_, err = db.Exec(table_schema)
-	if err != nil {
-		return fmt.Errorf("error in creating usage table: %v", err.Error())
-	}
-	return nil
-}

frontend/i18-generate-hash.js

@@ -1,24 +1,20 @@
-/* eslint-disable  */
-// @ts-ignore
-// @ts-nocheck
-
 const crypto = require('crypto');
 const fs = require('fs');
 const glob = require('glob');
 
 function generateChecksum(str, algorithm, encoding) {
-  return crypto
-    .createHash(algorithm || 'md5')
-    .update(str, 'utf8')
-    .digest(encoding || 'hex');
+	return crypto
+		.createHash(algorithm || 'md5')
+		.update(str, 'utf8')
+		.digest(encoding || 'hex');
 }
 
 const result = {};
 
-glob.sync(`public/locales/**/*.json`).forEach(path => {
-  const [_, lang] = path.split('public/locales');
-  const content = fs.readFileSync(path, { encoding: 'utf-8' });
-  result[lang.replace('.json', '')] = generateChecksum(content);
+glob.sync(`public/locales/**/*.json`).forEach((path) => {
+	const [_, lang] = path.split('public/locales');
+	const content = fs.readFileSync(path, { encoding: 'utf-8' });
+	result[lang.replace('.json', '')] = generateChecksum(content);
 });
 
 fs.writeFileSync('./i18n-translations-hash.json', JSON.stringify(result));

frontend/public/locales/en-GB/alerts.json

@@ -28,6 +28,7 @@
   "condition_required": "at least one metric condition is required",
   "alertname_required": "alert name is required",
   "promql_required": "promql expression is required when query format is set to PromQL",
+  "chquery_required": "query is required when query format is set to ClickHouse",
   "button_savechanges": "Save Rule",
   "button_createrule": "Create Rule",
   "button_returntorules": "Return to rules",
@@ -55,6 +56,7 @@
   "button_formula": "Formula",
   "tab_qb": "Query Builder",
   "tab_promql": "PromQL",
+  "tab_chquery": "ClickHouse Query",
   "title_confirm": "Confirm",
   "button_ok": "Yes",
   "button_cancel": "No",
@@ -88,5 +90,23 @@
   "user_guide_pql_step3": "Step 3 -Alert Configuration",
   "user_guide_pql_step3a": "Set alert severity, name and descriptions",
   "user_guide_pql_step3b": "Add tags to the alert in the Label field if needed",
-  "user_tooltip_more_help": "More details on how to create alerts"
+  "user_guide_ch_step1": "Step 1 - Define the metric",
+  "user_guide_ch_step1a": "Write a Clickhouse query for alert evaluation. Follow <0>this tutorial</0> to learn about query format and supported vars.",
+  "user_guide_ch_step1b": "Format the legends based on labels you want to highlight in the preview chart",
+  "user_guide_ch_step2": "Step 2 - Define Alert Conditions",
+  "user_guide_ch_step2a": "Select the threshold type and whether you want to alert above/below a value",
+  "user_guide_ch_step2b": "Enter the Alert threshold",
+  "user_guide_ch_step3": "Step 3 -Alert Configuration",
+  "user_guide_ch_step3a": "Set alert severity, name and descriptions",
+  "user_guide_ch_step3b": "Add tags to the alert in the Label field if needed",
+  "user_tooltip_more_help": "More details on how to create alerts",
+  "choose_alert_type": "Choose a type for the alert:",
+  "metric_based_alert": "Metric based Alert",
+  "metric_based_alert_desc": "Send a notification when a condition occurs in the metric data",
+  "log_based_alert": "Log-based Alert",
+  "log_based_alert_desc": "Send a notification when a condition occurs in the logs data.",
+  "traces_based_alert": "Trace-based Alert",
+  "traces_based_alert_desc": "Send a notification when a condition occurs in the traces data.",
+  "exceptions_based_alert": "Exceptions-based Alert",
+  "exceptions_based_alert_desc": "Send a notification when a condition occurs in the exceptions data."
 }

frontend/public/locales/en-GB/dashboard.json

@@ -1,6 +1,7 @@
 {
 	"create_dashboard": "Create Dashboard",
 	"import_json": "Import JSON",
+	"import_grafana_json": "Import Grafana JSON",
 	"copy_to_clipboard": "Copy To ClipBoard",
 	"download_json": "Download JSON",
 	"view_json": "View JSON",

frontend/public/locales/en-GB/licenses.json

@@ -9,5 +9,5 @@
   "tab_license_history": "History",
   "loading_licenses": "Loading licenses...",
   "enter_license_key": "Please enter a license key",
-  "license_applied": "License applied successfully, please refresh the page to see changes."
-}
+  "license_applied": "License applied successfully"
+}

frontend/public/locales/en-GB/traceDetails.json

@@ -0,0 +1,3 @@
+{
+	"search_tags": "Search Tag Names"
+}

frontend/public/locales/en/alerts.json

@@ -28,6 +28,7 @@
   "condition_required": "at least one metric condition is required",
   "alertname_required": "alert name is required",
   "promql_required": "promql expression is required when query format is set to PromQL",
+  "chquery_required": "query is required when query format is set to ClickHouse",
   "button_savechanges": "Save Rule",
   "button_createrule": "Create Rule",
   "button_returntorules": "Return to rules",
@@ -55,6 +56,7 @@
   "button_formula": "Formula",
   "tab_qb": "Query Builder",
   "tab_promql": "PromQL",
+  "tab_chquery": "ClickHouse Query",
   "title_confirm": "Confirm",
   "button_ok": "Yes",
   "button_cancel": "No",
@@ -88,5 +90,23 @@
   "user_guide_pql_step3": "Step 3 -Alert Configuration",
   "user_guide_pql_step3a": "Set alert severity, name and descriptions",
   "user_guide_pql_step3b": "Add tags to the alert in the Label field if needed",
-  "user_tooltip_more_help": "More details on how to create alerts"
+  "user_guide_ch_step1": "Step 1 - Define the metric",
+  "user_guide_ch_step1a": "Write a Clickhouse query for alert evaluation. Follow <0>this tutorial</0> to learn about query format and supported vars.",
+  "user_guide_ch_step1b": "Format the legends based on labels you want to highlight in the preview chart",
+  "user_guide_ch_step2": "Step 2 - Define Alert Conditions",
+  "user_guide_ch_step2a": "Select the threshold type and whether you want to alert above/below a value",
+  "user_guide_ch_step2b": "Enter the Alert threshold",
+  "user_guide_ch_step3": "Step 3 -Alert Configuration",
+  "user_guide_ch_step3a": "Set alert severity, name and descriptions",
+  "user_guide_ch_step3b": "Add tags to the alert in the Label field if needed",
+  "user_tooltip_more_help": "More details on how to create alerts",
+  "choose_alert_type": "Choose a type for the alert:",
+  "metric_based_alert": "Metric based Alert",
+  "metric_based_alert_desc": "Send a notification when a condition occurs in the metric data",
+  "log_based_alert": "Log-based Alert",
+  "log_based_alert_desc": "Send a notification when a condition occurs in the logs data.",
+  "traces_based_alert": "Trace-based Alert",
+  "traces_based_alert_desc": "Send a notification when a condition occurs in the traces data.",
+  "exceptions_based_alert": "Exceptions-based Alert",
+  "exceptions_based_alert_desc": "Send a notification when a condition occurs in the exceptions data."
 }

frontend/public/locales/en/dashboard.json

@@ -1,6 +1,7 @@
 {
 	"create_dashboard": "Create Dashboard",
 	"import_json": "Import JSON",
+	"import_grafana_json": "Import Grafana JSON",
 	"copy_to_clipboard": "Copy To ClipBoard",
 	"download_json": "Download JSON",
 	"view_json": "View JSON",

frontend/public/locales/en/licenses.json

@@ -9,5 +9,5 @@
   "tab_license_history": "History",
   "loading_licenses": "Loading licenses...",
   "enter_license_key": "Please enter a license key",
-  "license_applied": "License applied successfully, please refresh the page to see changes."
-}
+  "license_applied": "License applied successfully"
+}

frontend/public/locales/en/traceDetails.json

@@ -0,0 +1,3 @@
+{
+	"search_tags": "Search Tag Names"
+}

frontend/public/locales/en/translation.json

@@ -6,7 +6,7 @@
 	"release_notes": "Release Notes",
 	"read_how_to_upgrade": "Read instructions on how to upgrade",
 	"latest_version_signoz": "You are running the latest version of SigNoz.",
-	"stale_version": "You are on an older version and may be losing out on the latest features we have shipped. We recommend to upgrade to the latest version",
+	"stale_version": "You are on an older version and may be missing out on the latest features we have shipped. We recommend to upgrade to the latest version",
 	"oops_something_went_wrong_version": "Oops.. facing issues with fetching updated version information",
 	"n_a": "N/A",
 	"routes": {

frontend/src/AppRoutes/utils.ts

@@ -57,6 +57,7 @@ const afterLogin = async (
 				profilePictureURL: payload.profilePictureURL,
 				userId: payload.id,
 				orgId: payload.orgId,
+				userFlags: payload.flags,
 			},
 		});
 

frontend/src/api/dashboard/create.ts

@@ -7,8 +7,9 @@ import { PayloadProps, Props } from 'types/api/dashboard/create';
 const create = async (
 	props: Props,
 ): Promise<SuccessResponse<PayloadProps> | ErrorResponse> => {
+	const url = props.uploadedGrafana ? '/dashboards/grafana' : '/dashboards';
 	try {
-		const response = await axios.post('/dashboards', {
+		const response = await axios.post(url, {
 			...props,
 		});
 

frontend/src/api/dynamicConfigs/getDynamicConfigs.ts

@@ -0,0 +1,24 @@
+import axios from 'api';
+import { ErrorResponseHandler } from 'api/ErrorResponseHandler';
+import { AxiosError } from 'axios';
+import { ErrorResponse, SuccessResponse } from 'types/api';
+import { PayloadProps } from 'types/api/dynamicConfigs/getDynamicConfigs';
+
+const getDynamicConfigs = async (): Promise<
+	SuccessResponse<PayloadProps> | ErrorResponse
+> => {
+	try {
+		const response = await axios.get(`/configs`);
+
+		return {
+			statusCode: 200,
+			error: null,
+			message: response.data.status,
+			payload: response.data.data,
+		};
+	} catch (error) {
+		return ErrorResponseHandler(error as AxiosError);
+	}
+};
+
+export default getDynamicConfigs;

frontend/src/api/logs/livetail.ts

@@ -4,14 +4,16 @@ import { ENVIRONMENT } from 'constants/env';
 import { LOCALSTORAGE } from 'constants/localStorage';
 import { EventSourcePolyfill } from 'event-source-polyfill';
 
-export const LiveTail = (queryParams: string): EventSourcePolyfill => {
-	const dict = {
-		headers: {
-			Authorization: `Bearer ${getLocalStorageKey(LOCALSTORAGE.AUTH_TOKEN)}`,
-		},
-	};
-	return new EventSourcePolyfill(
+// 10 min in ms
+const TIMEOUT_IN_MS = 10 * 60 * 1000;
+
+export const LiveTail = (queryParams: string): EventSourcePolyfill =>
+	new EventSourcePolyfill(
 		`${ENVIRONMENT.baseURL}${apiV1}logs/tail?${queryParams}`,
-		dict,
+		{
+			headers: {
+				Authorization: `Bearer ${getLocalStorageKey(LOCALSTORAGE.AUTH_TOKEN)}`,
+			},
+			heartbeatTimeout: TIMEOUT_IN_MS,
+		},
 	);
-};

frontend/src/api/trace/getTraceItem.ts

@@ -1,15 +1,20 @@
 import axios from 'api';
 import { ErrorResponseHandler } from 'api/ErrorResponseHandler';
 import { AxiosError } from 'axios';
+import { formUrlParams } from 'container/TraceDetail/utils';
 import { ErrorResponse, SuccessResponse } from 'types/api';
-import { PayloadProps, Props } from 'types/api/trace/getTraceItem';
+import { GetTraceItemProps, PayloadProps } from 'types/api/trace/getTraceItem';
 
 const getTraceItem = async (
-	props: Props,
+	props: GetTraceItemProps,
 ): Promise<SuccessResponse<PayloadProps> | ErrorResponse> => {
 	try {
 		const response = await axios.request<PayloadProps>({
-			url: `/traces/${props.id}`,
+			url: `/traces/${props.id}${formUrlParams({
+				spanId: props.spanId,
+				levelUp: props.levelUp,
+				levelDown: props.levelDown,
+			})}`,
 			method: 'get',
 		});
 

frontend/src/api/user/setFlags.ts

@@ -0,0 +1,26 @@
+import axios from 'api';
+import { ErrorResponseHandler } from 'api/ErrorResponseHandler';
+import { AxiosError } from 'axios';
+import { ErrorResponse, SuccessResponse } from 'types/api';
+import { PayloadProps, Props } from 'types/api/user/setFlags';
+
+const setFlags = async (
+	props: Props,
+): Promise<SuccessResponse<PayloadProps> | ErrorResponse> => {
+	try {
+		const response = await axios.patch(`/user/${props.userId}/flags`, {
+			...props.flags,
+		});
+
+		return {
+			statusCode: 200,
+			error: null,
+			message: response.data?.status,
+			payload: response.data,
+		};
+	} catch (error) {
+		return ErrorResponseHandler(error as AxiosError);
+	}
+};
+
+export default setFlags;

frontend/src/api/utils.ts

@@ -41,6 +41,7 @@ export const Logout = (): void => {
 			orgName: '',
 			profilePictureURL: '',
 			userId: '',
+			userFlags: {},
 		},
 	});
 

frontend/src/components/Graph/Plugin/Legend.ts

@@ -62,6 +62,8 @@ export const legend = (id: string, isLonger: boolean): Plugin<ChartType> => {
 				li.style.marginTop = '5px';
 
 				li.onclick = (): void => {
+					// eslint-disable-next-line @typescript-eslint/ban-ts-comment
+					// @ts-ignore
 					const { type } = chart.config;
 					if (type === 'pie' || type === 'doughnut') {
 						// Pie and doughnut charts only have a single dataset and visibility is per item

frontend/src/components/Graph/xAxisConfig.ts

@@ -4,9 +4,6 @@ import { useSelector } from 'react-redux';
 import { AppState } from 'store/reducers';
 import { GlobalReducer } from 'types/reducer/globalTime';
 
-interface ITimeUnit {
-	[key: string]: TimeUnit;
-}
 interface IAxisTimeUintConfig {
 	unitName: TimeUnit;
 	multiplier: number;
@@ -22,7 +19,7 @@ export interface ITimeRange {
 	maxTime: number | null;
 }
 
-export const TIME_UNITS: ITimeUnit = {
+export const TIME_UNITS: Record<TimeUnit, TimeUnit> = {
 	millisecond: 'millisecond',
 	second: 'second',
 	minute: 'minute',
@@ -31,6 +28,7 @@ export const TIME_UNITS: ITimeUnit = {
 	week: 'week',
 	month: 'month',
 	year: 'year',
+	quarter: 'quarter',
 };
 
 const TIME_UNITS_CONFIG: IAxisTimeUintConfig[] = [
@@ -93,6 +91,7 @@ export const convertTimeRange = (
 	} catch (error) {
 		console.error(error);
 	}
+
 	return {
 		unitName: relevantTimeUnit.unitName,
 		stepSize: Math.floor(stepSize) || 1,

frontend/src/components/Logs/AddToQueryHOC.tsx

@@ -1,46 +1,21 @@
 import { Button, Popover } from 'antd';
-import getStep from 'lib/getStep';
 import { generateFilterQuery } from 'lib/logs/generateFilterQuery';
 import React, { memo, useCallback, useMemo } from 'react';
-import { connect, useDispatch, useSelector } from 'react-redux';
-import { bindActionCreators, Dispatch } from 'redux';
-import { ThunkDispatch } from 'redux-thunk';
-import { getLogs } from 'store/actions/logs/getLogs';
-import { getLogsAggregate } from 'store/actions/logs/getLogsAggregate';
+import { useDispatch, useSelector } from 'react-redux';
 import { AppState } from 'store/reducers';
-import AppActions from 'types/actions';
-import { SET_SEARCH_QUERY_STRING, TOGGLE_LIVE_TAIL } from 'types/actions/logs';
-import { GlobalReducer } from 'types/reducer/globalTime';
+import { SET_SEARCH_QUERY_STRING } from 'types/actions/logs';
 import { ILogsReducer } from 'types/reducer/logs';
 
-interface AddToQueryHOCProps {
-	fieldKey: string;
-	fieldValue: string;
-	children: React.ReactNode;
-	getLogs: (props: Parameters<typeof getLogs>[0]) => ReturnType<typeof getLogs>;
-	getLogsAggregate: (
-		props: Parameters<typeof getLogsAggregate>[0],
-	) => ReturnType<typeof getLogsAggregate>;
-}
 function AddToQueryHOC({
 	fieldKey,
 	fieldValue,
 	children,
-	getLogs,
-	getLogsAggregate,
 }: AddToQueryHOCProps): JSX.Element {
 	const {
 		searchFilter: { queryString },
-		logLinesPerPage,
-		idStart,
-		idEnd,
-		liveTail,
 	} = useSelector<AppState, ILogsReducer>((store) => store.logs);
 	const dispatch = useDispatch();
 
-	const { maxTime, minTime } = useSelector<AppState, GlobalReducer>(
-		(state) => state.globalTime,
-	);
 	const generatedQuery = useMemo(
 		() => generateFilterQuery({ fieldKey, fieldValue, type: 'IN' }),
 		[fieldKey, fieldValue],
@@ -58,69 +33,14 @@ function AddToQueryHOC({
 			type: SET_SEARCH_QUERY_STRING,
 			payload: updatedQueryString,
 		});
-		if (liveTail === 'STOPPED') {
-			getLogs({
-				q: updatedQueryString,
-				limit: logLinesPerPage,
-				orderBy: 'timestamp',
-				order: 'desc',
-				timestampStart: minTime,
-				timestampEnd: maxTime,
-				...(idStart ? { idGt: idStart } : {}),
-				...(idEnd ? { idLt: idEnd } : {}),
-			});
-			getLogsAggregate({
-				timestampStart: minTime,
-				timestampEnd: maxTime,
-				step: getStep({
-					start: minTime,
-					end: maxTime,
-					inputFormat: 'ns',
-				}),
-				q: updatedQueryString,
-				...(idStart ? { idGt: idStart } : {}),
-				...(idEnd ? { idLt: idEnd } : {}),
-			});
-		} else if (liveTail === 'PLAYING') {
-			dispatch({
-				type: TOGGLE_LIVE_TAIL,
-				payload: 'PAUSED',
-			});
-			setTimeout(
-				() =>
-					dispatch({
-						type: TOGGLE_LIVE_TAIL,
-						payload: liveTail,
-					}),
-				0,
-			);
-		}
-		// eslint-disable-next-line react-hooks/exhaustive-deps
-	}, [
-		dispatch,
-		generatedQuery,
-		getLogs,
-		idEnd,
-		idStart,
-		logLinesPerPage,
-		maxTime,
-		minTime,
-		queryString,
+	}, [dispatch, generatedQuery, queryString]);
+
+	const popOverContent = useMemo(() => <span>Add to query: {fieldKey}</span>, [
+		fieldKey,
 	]);
 
-	const popOverContent = (
-		<span style={{ fontSize: '0.9rem' }}>Add to query: {fieldKey}</span>
-	);
 	return (
-		<Button
-			size="small"
-			type="text"
-			style={{
-				margin: 0,
-				padding: 0,
-			}}
-			onClick={handleQueryAdd}
-		>
+		<Button size="small" type="text" onClick={handleQueryAdd}>
 			<Popover placement="top" content={popOverContent}>
 				{children}
 			</Popover>
@@ -128,20 +48,10 @@ function AddToQueryHOC({
 	);
 }
 
-interface DispatchProps {
-	getLogs: (
-		props: Parameters<typeof getLogs>[0],
-	) => (dispatch: Dispatch<AppActions>) => void;
-	getLogsAggregate: (
-		props: Parameters<typeof getLogsAggregate>[0],
-	) => (dispatch: Dispatch<AppActions>) => void;
+interface AddToQueryHOCProps {
+	fieldKey: string;
+	fieldValue: string;
+	children: React.ReactNode;
 }
 
-const mapDispatchToProps = (
-	dispatch: ThunkDispatch<unknown, unknown, AppActions>,
-): DispatchProps => ({
-	getLogs: bindActionCreators(getLogs, dispatch),
-	getLogsAggregate: bindActionCreators(getLogsAggregate, dispatch),
-});
-
-export default connect(null, mapDispatchToProps)(memo(AddToQueryHOC));
+export default memo(AddToQueryHOC);

frontend/src/components/MessageTip/index.tsx

@@ -0,0 +1,27 @@
+import React from 'react';
+
+import { StyledAlert } from './styles';
+
+interface MessageTipProps {
+	show?: boolean;
+	message: React.ReactNode | string;
+	action: React.ReactNode | undefined;
+}
+
+function MessageTip({
+	show,
+	message,
+	action,
+}: MessageTipProps): JSX.Element | null {
+	if (!show) return null;
+
+	return (
+		<StyledAlert showIcon description={message} type="info" action={action} />
+	);
+}
+
+MessageTip.defaultProps = {
+	show: false,
+};
+
+export default MessageTip;

frontend/src/components/MessageTip/styles.ts

@@ -0,0 +1,6 @@
+import { Alert } from 'antd';
+import styled from 'styled-components';
+
+export const StyledAlert = styled(Alert)`
+	align-items: center;
+`;

frontend/src/components/NotFound/constant.ts

@@ -0,0 +1 @@
+export const defaultText = 'Ah, seems like we reached a dead end!';

frontend/src/components/NotFound/index.tsx

@@ -2,45 +2,52 @@ import getLocalStorageKey from 'api/browser/localstorage/get';
 import NotFoundImage from 'assets/NotFound';
 import { LOCALSTORAGE } from 'constants/localStorage';
 import ROUTES from 'constants/routes';
-import React from 'react';
+import React, { useCallback } from 'react';
 import { useDispatch } from 'react-redux';
 import { Dispatch } from 'redux';
 import AppActions from 'types/actions';
 import { LOGGED_IN } from 'types/actions/app';
 
+import { defaultText } from './constant';
 import { Button, Container, Text, TextContainer } from './styles';
 
-function NotFound(): JSX.Element {
+function NotFound({ text = defaultText }: Props): JSX.Element {
 	const dispatch = useDispatch<Dispatch<AppActions>>();
 	const isLoggedIn = getLocalStorageKey(LOCALSTORAGE.IS_LOGGED_IN);
 
+	const onClickHandler = useCallback(() => {
+		if (isLoggedIn) {
+			dispatch({
+				type: LOGGED_IN,
+				payload: {
+					isLoggedIn: true,
+				},
+			});
+		}
+	}, [dispatch, isLoggedIn]);
+
 	return (
 		<Container>
 			<NotFoundImage />
 
 			<TextContainer>
-				<Text>Ah, seems like we reached a dead end!</Text>
+				<Text>{text}</Text>
 				<Text>Page Not Found</Text>
 			</TextContainer>
 
-			<Button
-				onClick={(): void => {
-					if (isLoggedIn) {
-						dispatch({
-							type: LOGGED_IN,
-							payload: {
-								isLoggedIn: true,
-							},
-						});
-					}
-				}}
-				to={ROUTES.APPLICATION}
-				tabIndex={0}
-			>
+			<Button onClick={onClickHandler} to={ROUTES.APPLICATION} tabIndex={0}>
 				Return To Services Page
 			</Button>
 		</Container>
 	);
 }
 
+interface Props {
+	text?: string;
+}
+
+NotFound.defaultProps = {
+	text: defaultText,
+};
+
 export default NotFound;

frontend/src/components/ReleaseNote/ReleaseNoteProps.ts

@@ -0,0 +1,4 @@
+export default interface ReleaseNoteProps {
+	path?: string;
+	release?: string;
+}

frontend/src/components/ReleaseNote/Releases/ReleaseNote0120.tsx

@@ -0,0 +1,73 @@
+import { Button, Space } from 'antd';
+import setFlags from 'api/user/setFlags';
+import MessageTip from 'components/MessageTip';
+import React, { useCallback } from 'react';
+import { useDispatch, useSelector } from 'react-redux';
+import { Dispatch } from 'redux';
+import { AppState } from 'store/reducers';
+import AppActions from 'types/actions';
+import { UPDATE_USER_FLAG } from 'types/actions/app';
+import { UserFlags } from 'types/api/user/setFlags';
+import AppReducer from 'types/reducer/app';
+
+import ReleaseNoteProps from '../ReleaseNoteProps';
+
+export default function ReleaseNote0120({
+	release,
+}: ReleaseNoteProps): JSX.Element | null {
+	const { user } = useSelector<AppState, AppReducer>((state) => state.app);
+
+	const dispatch = useDispatch<Dispatch<AppActions>>();
+
+	const handleDontShow = useCallback(async (): Promise<void> => {
+		const flags: UserFlags = { ReleaseNote0120Hide: 'Y' };
+
+		try {
+			dispatch({
+				type: UPDATE_USER_FLAG,
+				payload: {
+					flags,
+				},
+			});
+			if (!user) {
+				// no user is set, so escape the routine
+				return;
+			}
+
+			const response = await setFlags({ userId: user?.userId, flags });
+
+			if (response.statusCode !== 200) {
+				console.log('failed to complete do not show status', response.error);
+			}
+		} catch (e) {
+			// here we do not nothing as the cost of error is minor,
+			// the user can switch the do no show option again in the further.
+			console.log('unexpected error: failed to complete do not show status', e);
+		}
+	}, [dispatch, user]);
+
+	return (
+		<MessageTip
+			show
+			message={
+				<div>
+					You are using {release} of SigNoz. We have introduced distributed setup in
+					v0.12.0 release. If you use or plan to use clickhouse queries in dashboard
+					or alerts, you might want to read about querying the new distributed tables{' '}
+					<a
+						href="https://signoz.io/docs/operate/migration/upgrade-0.12/#querying-distributed-tables"
+						target="_blank"
+						rel="noreferrer"
+					>
+						here
+					</a>
+				</div>
+			}
+			action={
+				<Space>
+					<Button onClick={handleDontShow}>Do not show again</Button>
+				</Space>
+			}
+		/>
+	);
+}

frontend/src/components/ReleaseNote/index.tsx

@@ -0,0 +1,66 @@
+import ReleaseNoteProps from 'components/ReleaseNote/ReleaseNoteProps';
+import ReleaseNote0120 from 'components/ReleaseNote/Releases/ReleaseNote0120';
+import ROUTES from 'constants/routes';
+import React from 'react';
+import { useSelector } from 'react-redux';
+import { AppState } from 'store/reducers';
+import { UserFlags } from 'types/api/user/setFlags';
+import AppReducer from 'types/reducer/app';
+
+interface ComponentMapType {
+	match: (
+		path: string | undefined,
+		version: string,
+		userFlags: UserFlags | null,
+	) => boolean;
+	component: ({ path, release }: ReleaseNoteProps) => JSX.Element | null;
+}
+
+const allComponentMap: ComponentMapType[] = [
+	{
+		match: (
+			path: string | undefined,
+			version: string,
+			userFlags: UserFlags | null,
+		): boolean => {
+			if (!path) {
+				return false;
+			}
+			const allowedPaths = [
+				ROUTES.LIST_ALL_ALERT,
+				ROUTES.APPLICATION,
+				ROUTES.ALL_DASHBOARD,
+			];
+			return (
+				userFlags?.ReleaseNote0120Hide !== 'Y' &&
+				allowedPaths.includes(path) &&
+				version.startsWith('v0.12')
+			);
+		},
+		component: ReleaseNote0120,
+	},
+];
+
+// ReleaseNote prints release specific warnings and notes that
+// user needs to be aware of before using the upgraded version.
+function ReleaseNote({ path }: ReleaseNoteProps): JSX.Element | null {
+	const { userFlags, currentVersion } = useSelector<AppState, AppReducer>(
+		(state) => state.app,
+	);
+
+	const c = allComponentMap.find((item) => {
+		return item.match(path, currentVersion, userFlags);
+	});
+
+	if (!c) {
+		return null;
+	}
+
+	return <c.component path={path} release={currentVersion} />;
+}
+
+ReleaseNote.defaultProps = {
+	path: '',
+};
+
+export default ReleaseNote;

frontend/src/components/ValueGraph/styles.ts

@@ -2,5 +2,6 @@ import { Typography } from 'antd';
 import styled from 'styled-components';
 
 export const Value = styled(Typography)`
-	font-size: 3rem;
+	font-size: 2.5vw;
+	text-align: center;
 `;

frontend/src/constants/app.ts

@@ -8,3 +8,8 @@ export const DEFAULT_AUTH0_APP_REDIRECTION_PATH = ROUTES.APPLICATION;
 
 export const IS_SIDEBAR_COLLAPSED = 'isSideBarCollapsed';
 export const INVITE_MEMBERS_HASH = '#invite-team-members';
+
+export const SIGNOZ_UPGRADE_PLAN_URL =
+	'https://upgrade.signoz.io/upgrade-from-app';
+
+export const DASHBOARD_TIME_IN_DURATION = 'refreshInterval';

frontend/src/container/AllError/constant.ts

@@ -0,0 +1,9 @@
+const DEFAULT_FILTER_VALUE = '';
+const EXCEPTION_TYPE_FILTER_NAME = 'exceptionType';
+const SERVICE_NAME_FILTER_NAME = 'serviceName';
+
+export {
+	DEFAULT_FILTER_VALUE,
+	EXCEPTION_TYPE_FILTER_NAME,
+	SERVICE_NAME_FILTER_NAME,
+};

frontend/src/container/AllError/index.tsx

@@ -1,12 +1,26 @@
-import { notification, Table, TableProps, Tooltip, Typography } from 'antd';
+import { SearchOutlined } from '@ant-design/icons';
+import {
+	Button,
+	Card,
+	Input,
+	notification,
+	Space,
+	Table,
+	TableProps,
+	Tooltip,
+	Typography,
+} from 'antd';
+import { ColumnType } from 'antd/es/table';
 import { ColumnsType } from 'antd/lib/table';
+import { FilterConfirmProps } from 'antd/lib/table/interface';
 import getAll from 'api/errors/getAll';
 import getErrorCounts from 'api/errors/getErrorCounts';
 import ROUTES from 'constants/routes';
 import dayjs from 'dayjs';
+import useUrlQuery from 'hooks/useUrlQuery';
 import createQueryParams from 'lib/createQueryParams';
 import history from 'lib/history';
-import React, { useEffect, useMemo } from 'react';
+import React, { useCallback, useEffect, useMemo } from 'react';
 import { useTranslation } from 'react-i18next';
 import { useQueries } from 'react-query';
 import { useSelector } from 'react-redux';
@@ -17,7 +31,11 @@ import { Exception, PayloadProps } from 'types/api/errors/getAll';
 import { GlobalReducer } from 'types/reducer/globalTime';
 
 import {
+	extractFilterValues,
+	getDefaultFilterValue,
 	getDefaultOrder,
+	getFilterString,
+	getFilterValues,
 	getNanoSeconds,
 	getOffSet,
 	getOrder,
@@ -30,15 +48,27 @@ function AllErrors(): JSX.Element {
 	const { maxTime, minTime, loading } = useSelector<AppState, GlobalReducer>(
 		(state) => state.globalTime,
 	);
-	const { search, pathname } = useLocation();
-	const params = useMemo(() => new URLSearchParams(search), [search]);
-
+	const { pathname } = useLocation();
+	const params = useUrlQuery();
 	const { t } = useTranslation(['common']);
-
-	const updatedOrder = getOrder(params.get(urlKey.order));
-	const getUpdatedOffset = getOffSet(params.get(urlKey.offset));
-	const getUpdatedParams = getOrderParams(params.get(urlKey.orderParam));
-	const getUpdatedPageSize = getUpdatePageSize(params.get(urlKey.pageSize));
+	const {
+		updatedOrder,
+		getUpdatedOffset,
+		getUpdatedParams,
+		getUpdatedPageSize,
+		getUpdatedExceptionType,
+		getUpdatedServiceName,
+	} = useMemo(
+		() => ({
+			updatedOrder: getOrder(params.get(urlKey.order)),
+			getUpdatedOffset: getOffSet(params.get(urlKey.offset)),
+			getUpdatedParams: getOrderParams(params.get(urlKey.orderParam)),
+			getUpdatedPageSize: getUpdatePageSize(params.get(urlKey.pageSize)),
+			getUpdatedExceptionType: getFilterString(params.get(urlKey.exceptionType)),
+			getUpdatedServiceName: getFilterString(params.get(urlKey.serviceName)),
+		}),
+		[params],
+	);
 
 	const updatedPath = useMemo(
 		() =>
@@ -47,6 +77,8 @@ function AllErrors(): JSX.Element {
 				offset: getUpdatedOffset,
 				orderParam: getUpdatedParams,
 				pageSize: getUpdatedPageSize,
+				exceptionType: getUpdatedExceptionType,
+				serviceName: getUpdatedServiceName,
 			})}`,
 		[
 			pathname,
@@ -54,6 +86,8 @@ function AllErrors(): JSX.Element {
 			getUpdatedOffset,
 			getUpdatedParams,
 			getUpdatedPageSize,
+			getUpdatedExceptionType,
+			getUpdatedServiceName,
 		],
 	);
 
@@ -68,6 +102,8 @@ function AllErrors(): JSX.Element {
 					limit: getUpdatedPageSize,
 					offset: getUpdatedOffset,
 					orderParam: getUpdatedParams,
+					exceptionType: getUpdatedExceptionType,
+					serviceName: getUpdatedServiceName,
 				}),
 			enabled: !loading,
 		},
@@ -93,11 +129,123 @@ function AllErrors(): JSX.Element {
 		<Typography>{dayjs(value).format('DD/MM/YYYY HH:mm:ss A')}</Typography>
 	);
 
+	const filterIcon = useCallback(() => <SearchOutlined />, []);
+
+	const handleSearch = useCallback(
+		(
+			confirm: (param?: FilterConfirmProps) => void,
+			filterValue: string,
+			filterKey: string,
+		): VoidFunction => (): void => {
+			const { exceptionFilterValue, serviceFilterValue } = getFilterValues(
+				getUpdatedServiceName || '',
+				getUpdatedExceptionType || '',
+				filterKey,
+				filterValue || '',
+			);
+			history.replace(
+				`${pathname}?${createQueryParams({
+					order: updatedOrder,
+					offset: getUpdatedOffset,
+					orderParam: getUpdatedParams,
+					pageSize: getUpdatedPageSize,
+					exceptionType: exceptionFilterValue,
+					serviceName: serviceFilterValue,
+				})}`,
+			);
+			confirm();
+		},
+		[
+			getUpdatedExceptionType,
+			getUpdatedOffset,
+			getUpdatedPageSize,
+			getUpdatedParams,
+			getUpdatedServiceName,
+			pathname,
+			updatedOrder,
+		],
+	);
+
+	const filterDropdownWrapper = useCallback(
+		({ setSelectedKeys, selectedKeys, confirm, placeholder, filterKey }) => {
+			return (
+				<Card size="small">
+					<Space align="start" direction="vertical">
+						<Input
+							placeholder={placeholder}
+							value={selectedKeys[0]}
+							onChange={(e): void =>
+								setSelectedKeys(e.target.value ? [e.target.value] : [])
+							}
+							allowClear
+							defaultValue={getDefaultFilterValue(
+								filterKey,
+								getUpdatedServiceName,
+								getUpdatedExceptionType,
+							)}
+							onPressEnter={handleSearch(confirm, selectedKeys[0], filterKey)}
+						/>
+						<Button
+							type="primary"
+							onClick={handleSearch(confirm, selectedKeys[0], filterKey)}
+							icon={<SearchOutlined />}
+							size="small"
+						>
+							Search
+						</Button>
+					</Space>
+				</Card>
+			);
+		},
+		[getUpdatedExceptionType, getUpdatedServiceName, handleSearch],
+	);
+
+	const onExceptionTypeFilter = useCallback(
+		(value, record: Exception): boolean => {
+			if (record.exceptionType && typeof value === 'string') {
+				return record.exceptionType.toLowerCase().includes(value.toLowerCase());
+			}
+			return false;
+		},
+		[],
+	);
+
+	const onApplicationTypeFilter = useCallback(
+		(value, record: Exception): boolean => {
+			if (record.serviceName && typeof value === 'string') {
+				return record.serviceName.toLowerCase().includes(value.toLowerCase());
+			}
+			return false;
+		},
+		[],
+	);
+
+	const getFilter = useCallback(
+		(
+			onFilter: ColumnType<Exception>['onFilter'],
+			placeholder: string,
+			filterKey: string,
+		): ColumnType<Exception> => ({
+			onFilter,
+			filterIcon,
+			filterDropdown: ({ confirm, selectedKeys, setSelectedKeys }): JSX.Element =>
+				filterDropdownWrapper({
+					setSelectedKeys,
+					selectedKeys,
+					confirm,
+					placeholder,
+					filterKey,
+				}),
+		}),
+		[filterIcon, filterDropdownWrapper],
+	);
+
 	const columns: ColumnsType<Exception> = [
 		{
 			title: 'Exception Type',
 			dataIndex: 'exceptionType',
 			key: 'exceptionType',
+			...getFilter(onExceptionTypeFilter, 'Search By Exception', 'exceptionType'),
 			render: (value, record): JSX.Element => (
 				<Tooltip overlay={(): JSX.Element => value}>
 					<Link
@@ -177,29 +325,39 @@ function AllErrors(): JSX.Element {
 				updatedOrder,
 				'serviceName',
 			),
+			...getFilter(
+				onApplicationTypeFilter,
+				'Search By Application',
+				'serviceName',
+			),
 		},
 	];
 
-	const onChangeHandler: TableProps<Exception>['onChange'] = (
-		paginations,
-		_,
-		sorter,
-	) => {
-		if (!Array.isArray(sorter)) {
-			const { pageSize = 0, current = 0 } = paginations;
-			const { columnKey = '', order } = sorter;
-			const updatedOrder = order === 'ascend' ? 'ascending' : 'descending';
-
-			history.replace(
-				`${pathname}?${createQueryParams({
-					order: updatedOrder,
-					offset: (current - 1) * pageSize,
-					orderParam: columnKey,
-					pageSize,
-				})}`,
-			);
-		}
-	};
+	const onChangeHandler: TableProps<Exception>['onChange'] = useCallback(
+		(paginations, filters, sorter) => {
+			if (!Array.isArray(sorter)) {
+				const { pageSize = 0, current = 0 } = paginations;
+				const { columnKey = '', order } = sorter;
+				const updatedOrder = order === 'ascend' ? 'ascending' : 'descending';
+				const params = new URLSearchParams(window.location.search);
+				const { exceptionType, serviceName } = extractFilterValues(filters, {
+					serviceName: getFilterString(params.get(urlKey.serviceName)),
+					exceptionType: getFilterString(params.get(urlKey.exceptionType)),
+				});
+				history.replace(
+					`${pathname}?${createQueryParams({
+						order: updatedOrder,
+						offset: (current - 1) * pageSize,
+						orderParam: columnKey,
+						pageSize,
+						exceptionType,
+						serviceName,
+					})}`,
+				);
+			}
+		},
+		[pathname],
+	);
 
 	return (
 		<Table

frontend/src/container/AllError/utils.ts

@@ -1,7 +1,13 @@
-import { SortOrder } from 'antd/lib/table/interface';
+import { FilterValue, SortOrder } from 'antd/lib/table/interface';
 import Timestamp from 'timestamp-nano';
 import { Order, OrderBy } from 'types/api/errors/getAll';
 
+import {
+	DEFAULT_FILTER_VALUE,
+	EXCEPTION_TYPE_FILTER_NAME,
+	SERVICE_NAME_FILTER_NAME,
+} from './constant';
+
 export const isOrder = (order: string | null): order is Order =>
 	!!(order === 'ascending' || order === 'descending');
 
@@ -10,6 +16,8 @@ export const urlKey = {
 	offset: 'offset',
 	orderParam: 'orderParam',
 	pageSize: 'pageSize',
+	exceptionType: 'exceptionType',
+	serviceName: 'serviceName',
 };
 
 export const isOrderParams = (orderBy: string | null): orderBy is OrderBy => {
@@ -77,7 +85,7 @@ export const getDefaultOrder = (
 export const getNanoSeconds = (date: string): string => {
 	return (
 		Math.floor(new Date(date).getTime() / 1e3).toString() +
-		Timestamp.fromString(date).getNano().toString()
+		String(Timestamp.fromString(date).getNano().toString()).padStart(9, '0')
 	);
 };
 
@@ -87,3 +95,94 @@ export const getUpdatePageSize = (pageSize: string | null): number => {
 	}
 	return 10;
 };
+
+export const getFilterString = (filter: string | null): string => {
+	if (filter) {
+		return filter;
+	}
+	return '';
+};
+
+export const getDefaultFilterValue = (
+	filterKey: string | null,
+	serviceName: string,
+	exceptionType: string,
+): string | undefined => {
+	let defaultValue: string | undefined;
+	switch (filterKey) {
+		case SERVICE_NAME_FILTER_NAME:
+			defaultValue = serviceName;
+			break;
+		case EXCEPTION_TYPE_FILTER_NAME:
+			defaultValue = exceptionType;
+			break;
+		default:
+			break;
+	}
+	return defaultValue;
+};
+
+export const getFilterValues = (
+	serviceName: string,
+	exceptionType: string,
+	filterKey: string,
+	filterValue: string,
+): { exceptionFilterValue: string; serviceFilterValue: string } => {
+	let serviceFilterValue = serviceName;
+	let exceptionFilterValue = exceptionType;
+	switch (filterKey) {
+		case EXCEPTION_TYPE_FILTER_NAME:
+			exceptionFilterValue = filterValue;
+			break;
+		case SERVICE_NAME_FILTER_NAME:
+			serviceFilterValue = filterValue;
+			break;
+		default:
+			break;
+	}
+	return { exceptionFilterValue, serviceFilterValue };
+};
+
+type FilterValues = { exceptionType: string; serviceName: string };
+
+const extractSingleFilterValue = (
+	filterName: string,
+	filters: Filter,
+): string => {
+	const filterValues = filters[filterName];
+
+	if (
+		!filterValues ||
+		!Array.isArray(filterValues) ||
+		filterValues.length === 0
+	) {
+		return DEFAULT_FILTER_VALUE;
+	}
+
+	return String(filterValues[0]);
+};
+
+type Filter = Record<string, FilterValue | null>;
+
+export const extractFilterValues = (
+	filters: Filter,
+	prefilledFilters: FilterValues,
+): FilterValues => {
+	const filterValues: FilterValues = {
+		exceptionType: prefilledFilters.exceptionType,
+		serviceName: prefilledFilters.serviceName,
+	};
+	if (filters[EXCEPTION_TYPE_FILTER_NAME]) {
+		filterValues.exceptionType = extractSingleFilterValue(
+			EXCEPTION_TYPE_FILTER_NAME,
+			filters,
+		);
+	}
+	if (filters[SERVICE_NAME_FILTER_NAME]) {
+		filterValues.serviceName = extractSingleFilterValue(
+			SERVICE_NAME_FILTER_NAME,
+			filters,
+		);
+	}
+	return filterValues;
+};

frontend/src/container/AppLayout/index.tsx

@@ -1,4 +1,5 @@
 import { notification } from 'antd';
+import getDynamicConfigs from 'api/dynamicConfigs/getDynamicConfigs';
 import getFeaturesFlags from 'api/features/getFeatureFlags';
 import getUserLatestVersion from 'api/user/getLatestVersion';
 import getUserVersion from 'api/user/getVersion';
@@ -14,6 +15,7 @@ import { Dispatch } from 'redux';
 import { AppState } from 'store/reducers';
 import AppActions from 'types/actions';
 import {
+	UPDATE_CONFIGS,
 	UPDATE_CURRENT_ERROR,
 	UPDATE_CURRENT_VERSION,
 	UPDATE_FEATURE_FLAGS,
@@ -33,6 +35,7 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 		getUserVersionResponse,
 		getUserLatestVersionResponse,
 		getFeaturesResponse,
+		getDynamicConfigsResponse,
 	] = useQueries([
 		{
 			queryFn: getUserVersion,
@@ -48,6 +51,10 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 			queryFn: getFeaturesFlags,
 			queryKey: 'getFeatureFlags',
 		},
+		{
+			queryFn: getDynamicConfigs,
+			queryKey: 'getDynamicConfigs',
+		},
 	]);
 
 	useEffect(() => {
@@ -65,11 +72,15 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 		if (getFeaturesResponse.status === 'idle') {
 			getFeaturesResponse.refetch();
 		}
+		if (getDynamicConfigsResponse.status === 'idle') {
+			getDynamicConfigsResponse.refetch();
+		}
 	}, [
 		getFeaturesResponse,
 		getUserLatestVersionResponse,
 		getUserVersionResponse,
 		isLoggedIn,
+		getDynamicConfigsResponse,
 	]);
 
 	const { children } = props;
@@ -78,6 +89,7 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 
 	const latestCurrentCounter = useRef(0);
 	const latestVersionCounter = useRef(0);
+	const latestConfigCounter = useRef(0);
 
 	useEffect(() => {
 		if (
@@ -170,6 +182,23 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 				},
 			});
 		}
+
+		if (
+			getDynamicConfigsResponse.isFetched &&
+			getDynamicConfigsResponse.isSuccess &&
+			getDynamicConfigsResponse.data &&
+			getDynamicConfigsResponse.data.payload &&
+			latestConfigCounter.current === 0
+		) {
+			latestConfigCounter.current = 1;
+
+			dispatch({
+				type: UPDATE_CONFIGS,
+				payload: {
+					configs: getDynamicConfigsResponse.data.payload,
+				},
+			});
+		}
 	}, [
 		dispatch,
 		isLoggedIn,
@@ -187,6 +216,9 @@ function AppLayout(props: AppLayoutProps): JSX.Element {
 		getFeaturesResponse.isFetched,
 		getFeaturesResponse.isSuccess,
 		getFeaturesResponse.data,
+		getDynamicConfigsResponse.data,
+		getDynamicConfigsResponse.isFetched,
+		getDynamicConfigsResponse.isSuccess,
 	]);
 
 	const isToDisplayLayout = isLoggedIn;

frontend/src/container/ConfigDropdown/Config/ErrorLink.tsx

@@ -0,0 +1,33 @@
+import React, { PureComponent } from 'react';
+
+interface State {
+	hasError: boolean;
+}
+
+interface Props {
+	children: JSX.Element;
+}
+
+class ErrorLink extends PureComponent<Props, State> {
+	constructor(props: Props) {
+		super(props);
+		this.state = { hasError: false };
+	}
+
+	static getDerivedStateFromError(): State {
+		return { hasError: true };
+	}
+
+	render(): JSX.Element {
+		const { children } = this.props;
+		const { hasError } = this.state;
+
+		if (hasError) {
+			return <div />;
+		}
+
+		return children;
+	}
+}
+
+export default ErrorLink;

frontend/src/container/ConfigDropdown/Config/Link.tsx

@@ -0,0 +1,23 @@
+import React from 'react';
+import { Link } from 'react-router-dom';
+
+function LinkContainer({ children, href }: LinkContainerProps): JSX.Element {
+	const isInternalLink = href.startsWith('/');
+
+	if (isInternalLink) {
+		return <Link to={href}>{children}</Link>;
+	}
+
+	return (
+		<a rel="noreferrer" target="_blank" href={href}>
+			{children}
+		</a>
+	);
+}
+
+interface LinkContainerProps {
+	children: React.ReactNode;
+	href: string;
+}
+
+export default LinkContainer;

frontend/src/container/ConfigDropdown/Config/index.tsx

@@ -0,0 +1,51 @@
+import { Menu, Space } from 'antd';
+import Spinner from 'components/Spinner';
+import React, { Suspense, useMemo } from 'react';
+import { useSelector } from 'react-redux';
+import { AppState } from 'store/reducers';
+import { ConfigProps } from 'types/api/dynamicConfigs/getDynamicConfigs';
+import AppReducer from 'types/reducer/app';
+
+import ErrorLink from './ErrorLink';
+import LinkContainer from './Link';
+
+function HelpToolTip({ config }: HelpToolTipProps): JSX.Element {
+	const sortedConfig = useMemo(
+		() => config.components.sort((a, b) => a.position - b.position),
+		[config.components],
+	);
+
+	const { isDarkMode } = useSelector<AppState, AppReducer>((state) => state.app);
+
+	return (
+		<Menu.ItemGroup>
+			{sortedConfig.map((item) => {
+				const iconName = `${isDarkMode ? item.darkIcon : item.lightIcon}`;
+
+				const Component = React.lazy(
+					() => import(`@ant-design/icons/es/icons/${iconName}.js`),
+				);
+				return (
+					<ErrorLink key={item.text + item.href}>
+						<Suspense fallback={<Spinner height="5vh" />}>
+							<Menu.Item>
+								<LinkContainer href={item.href}>
+									<Space size="small" align="start">
+										<Component />
+										{item.text}
+									</Space>
+								</LinkContainer>
+							</Menu.Item>
+						</Suspense>
+					</ErrorLink>
+				);
+			})}
+		</Menu.ItemGroup>
+	);
+}
+
+interface HelpToolTipProps {
+	config: ConfigProps;
+}
+
+export default HelpToolTip;

frontend/src/container/ConfigDropdown/index.tsx

@@ -0,0 +1,67 @@
+import {
+	CaretDownFilled,
+	CaretUpFilled,
+	QuestionCircleFilled,
+	QuestionCircleOutlined,
+} from '@ant-design/icons';
+import { Dropdown, Menu, Space } from 'antd';
+import React, { useMemo, useState } from 'react';
+import { useSelector } from 'react-redux';
+import { AppState } from 'store/reducers';
+import AppReducer from 'types/reducer/app';
+
+import HelpToolTip from './Config';
+
+function DynamicConfigDropdown({
+	frontendId,
+}: DynamicConfigDropdownProps): JSX.Element {
+	const { configs, isDarkMode } = useSelector<AppState, AppReducer>(
+		(state) => state.app,
+	);
+	const [isHelpDropDownOpen, setIsHelpDropDownOpen] = useState<boolean>(false);
+
+	const config = useMemo(
+		() =>
+			Object.values(configs).find(
+				(config) => config.frontendPositionId === frontendId,
+			),
+		[frontendId, configs],
+	);
+
+	const onToggleHandler = (): void => {
+		setIsHelpDropDownOpen(!isHelpDropDownOpen);
+	};
+
+	if (!config) {
+		return <div />;
+	}
+
+	const Icon = isDarkMode ? QuestionCircleOutlined : QuestionCircleFilled;
+	const DropDownIcon = isHelpDropDownOpen ? CaretUpFilled : CaretDownFilled;
+
+	return (
+		<Dropdown
+			onVisibleChange={onToggleHandler}
+			trigger={['click']}
+			overlay={
+				<Menu>
+					<HelpToolTip config={config} />
+				</Menu>
+			}
+			visible={isHelpDropDownOpen}
+		>
+			<Space align="center">
+				<Icon
+					style={{ fontSize: 26, color: 'white', paddingTop: 26, cursor: 'pointer' }}
+				/>
+				<DropDownIcon style={{ color: 'white' }} />
+			</Space>
+		</Dropdown>
+	);
+}
+
+interface DynamicConfigDropdownProps {
+	frontendId: string;
+}
+
+export default DynamicConfigDropdown;

frontend/src/container/CreateAlertRule/SelectAlertType/index.tsx

@@ -0,0 +1,68 @@
+import { Row } from 'antd';
+import React from 'react';
+import { useTranslation } from 'react-i18next';
+import { AlertTypes } from 'types/api/alerts/alertTypes';
+
+import { AlertTypeCard, SelectTypeContainer } from './styles';
+
+interface OptionType {
+	title: string;
+	selection: AlertTypes;
+	description: string;
+}
+
+function SelectAlertType({ onSelect }: SelectAlertTypeProps): JSX.Element {
+	const { t } = useTranslation(['alerts']);
+
+	const renderOptions = (): JSX.Element => {
+		const optionList: OptionType[] = [
+			{
+				title: t('metric_based_alert'),
+				selection: AlertTypes.METRICS_BASED_ALERT,
+				description: t('metric_based_alert_desc'),
+			},
+			{
+				title: t('log_based_alert'),
+				selection: AlertTypes.LOGS_BASED_ALERT,
+				description: t('log_based_alert_desc'),
+			},
+			{
+				title: t('traces_based_alert'),
+				selection: AlertTypes.TRACES_BASED_ALERT,
+				description: t('traces_based_alert_desc'),
+			},
+			{
+				title: t('exceptions_based_alert'),
+				selection: AlertTypes.EXCEPTIONS_BASED_ALERT,
+				description: t('exceptions_based_alert_desc'),
+			},
+		];
+		return (
+			<>
+				{optionList.map((o: OptionType) => (
+					<AlertTypeCard
+						key={o.selection}
+						title={o.title}
+						onClick={(): void => {
+							onSelect(o.selection);
+						}}
+					>
+						{o.description}
+					</AlertTypeCard>
+				))}
+			</>
+		);
+	};
+	return (
+		<SelectTypeContainer>
+			<h3> {t('choose_alert_type')} </h3>
+			<Row>{renderOptions()}</Row>
+		</SelectTypeContainer>
+	);
+}
+
+interface SelectAlertTypeProps {
+	onSelect: (typ: AlertTypes) => void;
+}
+
+export default SelectAlertType;

frontend/src/container/CreateAlertRule/SelectAlertType/styles.ts

@@ -0,0 +1,16 @@
+import { Card } from 'antd';
+import styled from 'styled-components';
+
+export const SelectTypeContainer = styled.div`
+	&&& {
+		padding: 1rem;
+	}
+`;
+
+export const AlertTypeCard = styled(Card)`
+	&&& {
+		margin: 5px;
+		width: 21rem;
+		cursor: pointer;
+	}
+`;

frontend/src/container/CreateAlertRule/defaults.ts

@@ -0,0 +1,186 @@
+import { AlertTypes } from 'types/api/alerts/alertTypes';
+import {
+	AlertDef,
+	defaultCompareOp,
+	defaultEvalWindow,
+	defaultMatchType,
+} from 'types/api/alerts/def';
+
+export const alertDefaults: AlertDef = {
+	alertType: AlertTypes.METRICS_BASED_ALERT,
+	condition: {
+		compositeMetricQuery: {
+			builderQueries: {
+				A: {
+					queryName: 'A',
+					name: 'A',
+					formulaOnly: false,
+					metricName: '',
+					tagFilters: {
+						op: 'AND',
+						items: [],
+					},
+					groupBy: [],
+					aggregateOperator: 1,
+					expression: 'A',
+					disabled: false,
+					toggleDisable: false,
+					toggleDelete: false,
+				},
+			},
+			promQueries: {},
+			chQueries: {},
+			queryType: 1,
+		},
+		op: defaultCompareOp,
+		matchType: defaultMatchType,
+	},
+	labels: {
+		severity: 'warning',
+	},
+	annotations: {
+		description: 'A new alert',
+	},
+	evalWindow: defaultEvalWindow,
+};
+
+export const logAlertDefaults: AlertDef = {
+	alertType: AlertTypes.LOGS_BASED_ALERT,
+	condition: {
+		compositeMetricQuery: {
+			builderQueries: {
+				A: {
+					queryName: 'A',
+					name: 'A',
+					formulaOnly: false,
+					metricName: '',
+					tagFilters: {
+						op: 'AND',
+						items: [],
+					},
+					groupBy: [],
+					aggregateOperator: 1,
+					expression: 'A',
+					disabled: false,
+					toggleDisable: false,
+					toggleDelete: false,
+				},
+			},
+			promQueries: {},
+			chQueries: {
+				A: {
+					name: 'A',
+					query: `select \ntoStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 MINUTE) AS interval, \ntoFloat64(count()) as value \nFROM signoz_logs.distributed_logs  \nWHERE timestamp BETWEEN {{.start_timestamp_nano}} AND {{.end_timestamp_nano}}  \nGROUP BY interval;\n\n-- available variables:\n-- \t{{.start_timestamp_nano}}\n-- \t{{.end_timestamp_nano}}\n\n-- required columns (or alias):\n-- \tvalue\n-- \tinterval`,
+					rawQuery: `select \ntoStartOfInterval(fromUnixTimestamp64Nano(timestamp), INTERVAL 30 MINUTE) AS interval, \ntoFloat64(count()) as value \nFROM signoz_logs.distributed_logs  \nWHERE timestamp BETWEEN {{.start_timestamp_nano}} AND {{.end_timestamp_nano}}  \nGROUP BY interval;\n\n-- available variables:\n-- \t{{.start_timestamp_nano}}\n-- \t{{.end_timestamp_nano}}\n\n-- required columns (or alias):\n-- \tvalue\n-- \tinterval`,
+					legend: '',
+					disabled: false,
+				},
+			},
+			queryType: 2,
+		},
+		op: defaultCompareOp,
+		matchType: '4',
+	},
+	labels: {
+		severity: 'warning',
+		details: `${window.location.protocol}//${window.location.host}/logs`,
+	},
+	annotations: {
+		description: 'A new log-based alert',
+	},
+	evalWindow: defaultEvalWindow,
+};
+
+export const traceAlertDefaults: AlertDef = {
+	alertType: AlertTypes.TRACES_BASED_ALERT,
+	condition: {
+		compositeMetricQuery: {
+			builderQueries: {
+				A: {
+					queryName: 'A',
+					name: 'A',
+					formulaOnly: false,
+					metricName: '',
+					tagFilters: {
+						op: 'AND',
+						items: [],
+					},
+					groupBy: [],
+					aggregateOperator: 1,
+					expression: 'A',
+					disabled: false,
+					toggleDisable: false,
+					toggleDelete: false,
+				},
+			},
+			promQueries: {},
+			chQueries: {
+				A: {
+					name: 'A',
+					rawQuery: `SELECT \n\ttoStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval, \n\ttagMap['peer.service'] AS op_name, \n\ttoFloat64(avg(durationNano)) AS value \nFROM signoz_traces.distributed_signoz_index_v2  \nWHERE tagMap['peer.service']!='' \nAND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}} \nGROUP BY (op_name, interval);\n\n-- available variables:\n-- \t{{.start_datetime}}\n-- \t{{.end_datetime}}\n\n-- required column alias:\n-- \tvalue\n-- \tinterval`,
+					query: `SELECT \n\ttoStartOfInterval(timestamp, INTERVAL 1 MINUTE) AS interval, \n\ttagMap['peer.service'] AS op_name, \n\ttoFloat64(avg(durationNano)) AS value \nFROM signoz_traces.distributed_signoz_index_v2  \nWHERE tagMap['peer.service']!='' \nAND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}} \nGROUP BY (op_name, interval);\n\n-- available variables:\n-- \t{{.start_datetime}}\n-- \t{{.end_datetime}}\n\n-- required column alias:\n-- \tvalue\n-- \tinterval`,
+					legend: '',
+					disabled: false,
+				},
+			},
+			queryType: 2,
+		},
+		op: defaultCompareOp,
+		matchType: '4',
+	},
+	labels: {
+		severity: 'warning',
+		details: `${window.location.protocol}//${window.location.host}/traces`,
+	},
+	annotations: {
+		description: 'A new trace-based alert',
+	},
+	evalWindow: defaultEvalWindow,
+};
+
+export const exceptionAlertDefaults: AlertDef = {
+	alertType: AlertTypes.EXCEPTIONS_BASED_ALERT,
+	condition: {
+		compositeMetricQuery: {
+			builderQueries: {
+				A: {
+					queryName: 'A',
+					name: 'A',
+					formulaOnly: false,
+					metricName: '',
+					tagFilters: {
+						op: 'AND',
+						items: [],
+					},
+					groupBy: [],
+					aggregateOperator: 1,
+					expression: 'A',
+					disabled: false,
+					toggleDisable: false,
+					toggleDelete: false,
+				},
+			},
+			promQueries: {},
+			chQueries: {
+				A: {
+					name: 'A',
+					rawQuery: `SELECT \n\tcount() as value,\n\ttoStartOfInterval(timestamp, toIntervalMinute(1)) AS interval,\n\tserviceName\nFROM signoz_traces.distributed_signoz_error_index_v2\nWHERE exceptionType !='OSError'\nAND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}\nGROUP BY serviceName, interval;\n\n-- available variables:\n-- \t{{.start_datetime}}\n-- \t{{.end_datetime}}\n\n-- required column alias:\n-- \tvalue\n-- \tinterval`,
+					query: `SELECT \n\tcount() as value,\n\ttoStartOfInterval(timestamp, toIntervalMinute(1)) AS interval,\n\tserviceName\nFROM signoz_traces.distributed_signoz_error_index_v2\nWHERE exceptionType !='OSError'\nAND timestamp BETWEEN {{.start_datetime}} AND {{.end_datetime}}\nGROUP BY serviceName, interval;\n\n-- available variables:\n-- \t{{.start_datetime}}\n-- \t{{.end_datetime}}\n\n-- required column alias:\n-- \tvalue\n-- \tinterval`,
+					legend: '',
+					disabled: false,
+				},
+			},
+			queryType: 2,
+		},
+		op: defaultCompareOp,
+		matchType: '4',
+	},
+	labels: {
+		severity: 'warning',
+		details: `${window.location.protocol}//${window.location.host}/exceptions`,
+	},
+	annotations: {
+		description: 'A new exceptions-based alert',
+	},
+	evalWindow: defaultEvalWindow,
+};

frontend/src/container/CreateAlertRule/index.tsx

@@ -1,22 +1,57 @@
-import { Form } from 'antd';
+import { Form, Row } from 'antd';
 import FormAlertRules from 'container/FormAlertRules';
-import React from 'react';
-import { AlertDef } from 'types/api/alerts/def';
+import React, { useState } from 'react';
+import { AlertTypes } from 'types/api/alerts/alertTypes';
 
-function CreateRules({ initialValue }: CreateRulesProps): JSX.Element {
+import {
+	alertDefaults,
+	exceptionAlertDefaults,
+	logAlertDefaults,
+	traceAlertDefaults,
+} from './defaults';
+import SelectAlertType from './SelectAlertType';
+
+function CreateRules(): JSX.Element {
+	const [initValues, setInitValues] = useState(alertDefaults);
+	const [step, setStep] = useState(0);
+	const [alertType, setAlertType] = useState<AlertTypes>(
+		AlertTypes.METRICS_BASED_ALERT,
+	);
 	const [formInstance] = Form.useForm();
 
+	const onSelectType = (typ: AlertTypes): void => {
+		setAlertType(typ);
+		switch (typ) {
+			case AlertTypes.LOGS_BASED_ALERT:
+				setInitValues(logAlertDefaults);
+				break;
+			case AlertTypes.TRACES_BASED_ALERT:
+				setInitValues(traceAlertDefaults);
+				break;
+			case AlertTypes.EXCEPTIONS_BASED_ALERT:
+				setInitValues(exceptionAlertDefaults);
+				break;
+			default:
+				setInitValues(alertDefaults);
+		}
+		setStep(1);
+	};
+
+	if (step === 0) {
+		return (
+			<Row wrap={false}>
+				<SelectAlertType onSelect={onSelectType} />
+			</Row>
+		);
+	}
 	return (
 		<FormAlertRules
+			alertType={alertType}
 			formInstance={formInstance}
-			initialValue={initialValue}
+			initialValue={initValues}
 			ruleId={0}
 		/>
 	);
 }
 
-interface CreateRulesProps {
-	initialValue: AlertDef;
-}
-
 export default CreateRules;

frontend/src/container/EditRules/index.tsx

@@ -1,6 +1,7 @@
 import { Form } from 'antd';
 import FormAlertRules from 'container/FormAlertRules';
 import React from 'react';
+import { AlertTypes } from 'types/api/alerts/alertTypes';
 import { AlertDef } from 'types/api/alerts/def';
 
 function EditRules({ initialValue, ruleId }: EditRulesProps): JSX.Element {
@@ -8,6 +9,11 @@ function EditRules({ initialValue, ruleId }: EditRulesProps): JSX.Element {
 
 	return (
 		<FormAlertRules
+			alertType={
+				initialValue.alertType
+					? (initialValue.alertType as AlertTypes)
+					: AlertTypes.METRICS_BASED_ALERT
+			}
 			formInstance={formInstance}
 			initialValue={initialValue}
 			ruleId={ruleId}

frontend/src/container/FormAlertRules/ChQuerySection/ChQuerySection.tsx

@@ -0,0 +1,46 @@
+import ClickHouseQueryBuilder from 'container/NewWidget/LeftContainer/QuerySection/QueryBuilder/clickHouse/query';
+import { IClickHouseQueryHandleChange } from 'container/NewWidget/LeftContainer/QuerySection/QueryBuilder/clickHouse/types';
+import React from 'react';
+import { IChQueries } from 'types/api/alerts/compositeQuery';
+
+import { rawQueryToIChQuery, toIClickHouseQuery } from './transform';
+
+function ChQuerySection({
+	chQueries,
+	setChQueries,
+}: ChQuerySectionProps): JSX.Element {
+	const handleChQueryChange = ({
+		rawQuery,
+		legend,
+		toggleDelete,
+	}: IClickHouseQueryHandleChange): void => {
+		const chQuery = rawQueryToIChQuery(
+			chQueries.A,
+			rawQuery,
+			legend,
+			toggleDelete,
+		);
+
+		setChQueries({
+			A: {
+				...chQuery,
+			},
+		});
+	};
+
+	return (
+		<ClickHouseQueryBuilder
+			key="A"
+			queryIndex="A"
+			queryData={toIClickHouseQuery(chQueries?.A)}
+			handleQueryChange={handleChQueryChange}
+		/>
+	);
+}
+
+interface ChQuerySectionProps {
+	chQueries: IChQueries;
+	setChQueries: (q: IChQueries) => void;
+}
+
+export default ChQuerySection;

frontend/src/container/FormAlertRules/ChQuerySection/index.ts

@@ -0,0 +1,3 @@
+import ChQuerySection from './ChQuerySection';
+
+export default ChQuerySection;

frontend/src/container/FormAlertRules/ChQuerySection/transform.ts

@@ -0,0 +1,37 @@
+import { IChQuery } from 'types/api/alerts/compositeQuery';
+import { IClickHouseQuery } from 'types/api/dashboard/getAll';
+
+// @description rawQueryToIChQuery transforms raw query (from ClickHouseQueryBuilder)
+// to alert specific IChQuery format
+export const rawQueryToIChQuery = (
+	src: IChQuery,
+	rawQuery: string | undefined,
+	legend: string | undefined,
+	toggleDelete: boolean | undefined,
+): IChQuery => {
+	if (toggleDelete) {
+		return {
+			rawQuery: '',
+			legend: '',
+			name: 'A',
+			disabled: false,
+			query: '',
+		};
+	}
+
+	return {
+		rawQuery: rawQuery !== undefined ? rawQuery : src.query,
+		query: rawQuery !== undefined ? rawQuery : src.query,
+		legend: legend !== undefined ? legend : src.legend,
+		name: 'A',
+		disabled: false,
+	};
+};
+
+// @description toIClickHouseQuery transforms IChQuery (alert specific) to
+// ClickHouseQueryBuilder format. The main difference is
+// use of rawQuery (in ClickHouseQueryBuilder)
+// and query (in alert builder)
+export const toIClickHouseQuery = (src: IChQuery): IClickHouseQuery => {
+	return { ...src, name: 'A', rawQuery: src.query };
+};

frontend/src/container/FormAlertRules/ChartPreview/index.tsx

@@ -1,11 +1,12 @@
 import { InfoCircleOutlined } from '@ant-design/icons';
 import { StaticLineProps } from 'components/Graph';
+import Spinner from 'components/Spinner';
 import GridGraphComponent from 'container/GridGraphComponent';
 import { GRAPH_TYPES } from 'container/NewDashboard/ComponentsSlider';
 import { timePreferenceType } from 'container/NewWidget/RightContainer/timeItems';
 import { Time } from 'container/TopNav/DateTimeSelection/config';
 import getChartData from 'lib/getChartData';
-import React from 'react';
+import React, { useMemo } from 'react';
 import { useTranslation } from 'react-i18next';
 import { useQuery } from 'react-query';
 import { GetMetricQueryRange } from 'store/actions/dashboard/getQueryResults';
@@ -22,6 +23,10 @@ export interface ChartPreviewProps {
 	selectedInterval?: Time;
 	headline?: JSX.Element;
 	threshold?: number | undefined;
+	userQueryKey?: string;
+}
+interface QueryResponseError {
+	message?: string;
 }
 
 function ChartPreview({
@@ -32,6 +37,7 @@ function ChartPreview({
 	selectedInterval = '5min',
 	headline,
 	threshold,
+	userQueryKey,
 }: ChartPreviewProps): JSX.Element | null {
 	const { t } = useTranslation('alerts');
 	const staticLine: StaticLineProps | undefined =
@@ -46,9 +52,34 @@ function ChartPreview({
 			  }
 			: undefined;
 
-	const queryKey = JSON.stringify(query);
+	const canQuery = useMemo((): boolean => {
+		if (!query || query == null) {
+			return false;
+		}
+
+		switch (query?.queryType) {
+			case EQueryType.PROM:
+				return query.promQL?.length > 0 && query.promQL[0].query !== '';
+			case EQueryType.CLICKHOUSE:
+				return (
+					query.clickHouse?.length > 0 && query.clickHouse[0].rawQuery?.length > 0
+				);
+			case EQueryType.QUERY_BUILDER:
+				return (
+					query.metricsBuilder?.queryBuilder?.length > 0 &&
+					query.metricsBuilder?.queryBuilder[0].metricName !== ''
+				);
+			default:
+				return false;
+		}
+	}, [query]);
+
 	const queryResponse = useQuery({
-		queryKey: ['chartPreview', queryKey, selectedInterval],
+		queryKey: [
+			'chartPreview',
+			userQueryKey || JSON.stringify(query),
+			selectedInterval,
+		],
 		queryFn: () =>
 			GetMetricQueryRange({
 				query: query || {
@@ -64,14 +95,8 @@ function ChartPreview({
 				graphType,
 				selectedTime,
 			}),
-		enabled:
-			query != null &&
-			((query.queryType === EQueryType.PROM &&
-				query.promQL?.length > 0 &&
-				query.promQL[0].query !== '') ||
-				(query.queryType === EQueryType.QUERY_BUILDER &&
-					query.metricsBuilder?.queryBuilder?.length > 0 &&
-					query.metricsBuilder?.queryBuilder[0].metricName !== '')),
+		retry: false,
+		enabled: canQuery,
 	});
 
 	const chartDataSet = queryResponse.isError
@@ -89,15 +114,14 @@ function ChartPreview({
 	return (
 		<ChartContainer>
 			{headline}
-			{(queryResponse?.data?.error || queryResponse?.isError) && (
+			{(queryResponse?.isError || queryResponse?.error) && (
 				<FailedMessageContainer color="red" title="Failed to refresh the chart">
 					<InfoCircleOutlined />{' '}
-					{queryResponse?.data?.error ||
-						queryResponse?.error ||
+					{(queryResponse?.error as QueryResponseError).message ||
 						t('preview_chart_unexpected_error')}
 				</FailedMessageContainer>
 			)}
-
+			{queryResponse.isLoading && <Spinner size="large" tip="Loading..." />}
 			{chartDataSet && !queryResponse.isError && (
 				<GridGraphComponent
 					title={name}
@@ -118,6 +142,7 @@ ChartPreview.defaultProps = {
 	selectedInterval: '5min',
 	headline: undefined,
 	threshold: undefined,
+	userQueryKey: '',
 };
 
 export default ChartPreview;