Updated codebase to use QuestDB instead of streaming files from S3 #2

mccaffers · 2025-11-14T18:17:39Z

No description provided.

Copilot

Pull Request Overview

This pull request migrates the backtesting codebase from streaming files from S3 to using QuestDB as a data source. The changes are extensive and involve significant architectural modifications.

Key Changes

Upgraded from .NET 8.0 to .NET 9.0 across all projects
Replaced S3-based file streaming with QuestDB direct database queries
Refactored environment variable management from mock-based to injection-based pattern
Removed web server and UI components
Updated test infrastructure to use JSON configuration files instead of mocks

Reviewed Changes

Copilot reviewed 150 out of 155 changed files in this pull request and generated 30 comments.

Show a summary per file

File	Description
tests/test.csproj	Upgraded target framework to net9.0
tests/resources/environmentVariables.json	Added JSON configuration file for test environment variables
tests/backtesting/TestEnvironmentSetup.cs	Refactored to load environment from JSON and use injection pattern
tests/backtesting/UtilitesTests/ExtensionTests.cs	Updated tests to use new environment setup
tests/backtesting/StrategyTests/RandomStrategyTests.cs	Modernized test implementation with proper AAA pattern
tests/backtesting/Shared.cs	Added EnvironmentJson class for JSON deserialization
tests/backtesting/ReportingTests/ReportingTests.cs	Updated tests to use environment injection
src/utilities/utilities.csproj	Upgraded to net9.0, added MathNet.Numerics package
src/utilities/StandardDeviation.cs	Added new statistical calculation utilities
src/utilities/ServiceExtension.cs	Removed environment variable dependency, added PropertyCopier utility
src/utilities/ReportEnvironmentVariables.cs	Added new utility for reporting environment variables
src/utilities/EnvironmentVariables_v2.cs	Implemented new enum-based environment variable system
src/utilities/ConsoleLogger.cs	Updated to use new environment variable system
src/utilities/MedianExtensions.cs	Added median calculation extensions
src/utilities/LinearRegression.cs	Added linear regression utilities
src/strategies/strategies.csproj	Upgraded to net9.0 and Microsoft.Extensions.Hosting 8.0.1
src/strategies/Random.cs	Updated to use new strategy definition pattern
src/strategies/BaseStrategy.cs	Removed environment variables and web notification dependencies
src/backtesting/backtesting-engine.csproj	Upgraded to net9.0, added Npgsql for QuestDB connectivity
src/backtesting/Program.cs	Major refactoring to use QuestDB instead of S3 streaming
src/backtesting/Ingest/IngestFromQuestDB.cs	New ingestion layer for QuestDB
src/backtesting/Operations/Positions.cs	Updated to use new environment variable system
src/backtesting/TradeManagement/CloseOrder.cs	Simplified trade closing logic
src/backtesting/Analysis/Reporting.cs	Enhanced reporting with statistical metrics
src/Models/models.csproj	Upgraded to net9.0, added MemoryPack serialization
src/Models/StrategyDefinition.cs	Added new strategy configuration model

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-14T18:22:40Z

tests/resources/environmentVariables.json

+    "LAMBDA_LOG": "FALSE",
+    "SYSTEM_LOG": "TRUE",
+    "CONSOLE_LOG": "TRUE",
+    "SCALING_FACRTOR": "TestEnvironmentSetup,1000;",


Spelling error: "SCALING_FACRTOR" should be "SCALING_FACTOR"

Copilot · 2025-11-14T18:22:40Z

tests/backtesting/Shared.cs

+
+    // TradingVariables
+    public required string STRATEGY { get; set; }  
+    public required string SCALING_FACRTOR  { get; set; }  


Spelling error: "SCALING_FACRTOR" should be "SCALING_FACTOR". This typo appears consistently throughout the codebase.

Copilot · 2025-11-14T18:22:40Z

src/utilities/EnvironmentVariables_v2.cs

+public enum TradingVariables
+{
+    STRATEGY,
+    SCALING_FACRTOR,


Spelling error: "SCALING_FACRTOR" should be "SCALING_FACTOR"

Copilot · 2025-11-14T18:22:40Z

src/utilities/LinearRegression.cs

+    /// <param name="exclusiveEnd">The exclusive exclusiveEnd index.</param>
+    /// <param name="rsquared">The r^2 value of the line.</param>
+    /// <param name="yintercept">The y-intercept value of the line (i.e. y = ax + b, yintercept is b).</param>
+    /// <param name="slope">The slop of the line (i.e. y = ax + b, slope is a).</param>


Spelling error in comment: "slop" should be "slope"

Copilot · 2025-11-14T18:22:41Z

src/backtesting/Program.cs

+        EnvironmentVariables.Inject(LoggingVariables.LAMBDA_LOG, BACKTESTING_VARIABLES.LAMBDA_LOG.ToString());
+        EnvironmentVariables.Inject(LoggingVariables.SYSTEM_LOG, BACKTESTING_VARIABLES.SYSTEM_LOG.ToString());
+
+        EnvironmentVariables.Inject(TradingVariables.SCALING_FACRTOR, "AUDUSD,10000;EURUSD,10000;GBRIDXGBP,1;GBPUSD,10000;NZDUSD,10000;USDJPY,100;GBPJPY,100;EURJPY,100;USDCAD,10000;FRAIDXEUR,1;EURGBP,10000;USA500IDXUSD,1;AUSIDXAUD,1;USDCHF,10000;XAUUSD,1;XAGUSD,1;USATECHIDXUSD,1;EURCHF,10000;DEUIDXEUR,1;USA30IDXUSD,1;LIGHTCMDUSD,1;JPNIDXJPY,1;BRENTCMDUSD,1;");


Spelling error: "SCALING_FACRTOR" should be "SCALING_FACTOR"

Copilot · 2025-11-14T18:22:46Z

src/utilities/EnvironmentVariables_v2.cs

+    }
+
+    private static IEnumerable<DictionaryEntry>? entries = Environment.GetEnvironmentVariables().Cast<DictionaryEntry>();
+    private static Dictionary<String, String> envCache = [];


Field 'envCache' can be 'readonly'.

Copilot · 2025-11-14T18:22:47Z

src/backtesting/Ingest/IngestFromQuestDB.cs

+        catch (Exception ex)
+        {
+            result = ex.Message;
+            Console.WriteLine(ex);
+            ConsoleLogger.Log($"Unexpected error: {result}");
+            await SendStackException(ex);
+
+        }


Generic catch clause.

Copilot · 2025-11-14T18:22:47Z

src/backtesting/Ingest/IngestFromQuestDB.cs

+        catch (Exception ex)
+        {
+            ConsoleLogger.Log($"Error during cleanup: {ex.Message}");
+        }


Generic catch clause.

Copilot · 2025-11-14T18:22:47Z

src/backtesting/Ingest/IngestFromQuestDB.cs

+        catch (Exception ex)
+        {
+            throw new TradingException($"Error retrieving OHLC data: {ex.Message}", ex);
+        }


Generic catch clause.

Copilot · 2025-11-14T18:22:48Z

src/backtesting/Operations/Positions.cs

+        if (tradingObjects.openTrades.Count > 0)
+        {
+            return tradingObjects.openTrades.Select(x => x.Value);
+        }
+        else
+        {
            return [];
        }


Both branches of this 'if' statement return - consider using '?' to express intent better.

Copilot

Pull Request Overview

Copilot reviewed 150 out of 155 changed files in this pull request and generated 2 comments.

💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.

Copilot · 2025-11-15T10:16:55Z

src/backtesting/Ingest/IngestFromQuestDB.cs

+            Host=localhost;
+            Port=8812;
+            Username=admin;
+            Password=quest;


Hardcoded database credentials are a security risk. The password 'quest' should be loaded from environment variables or a secure configuration source instead of being hardcoded in the connection string.

Copilot · 2025-11-15T10:16:55Z

src/backtesting/Ingest/IngestFromQuestDB.cs

+            count++; // to track results
+            if (count % 5 != 0)
+            {
+                continue; // skip every 10th tick


The comment says "skip every 10th tick" but the code actually skips 4 out of every 5 ticks (only processes when count % 5 == 0). This comment is misleading and should be corrected to accurately reflect the filtering logic.

sonarqubecloud · 2025-11-30T09:38:47Z

Quality Gate failed

Failed conditions
2 Security Hotspots
33.6% Coverage on New Code (required ≥ 80%)
C Reliability Rating on New Code (required ≥ A)

See analysis details on SonarQube Cloud

Catch issues before they fail your Quality Gate with our IDE extension SonarQube for IDE

Updated codebase to use QuestDB instead of streaming files from S3

32a19fe

Copilot AI review requested due to automatic review settings November 14, 2025 18:17

mccaffers linked an issue Nov 14, 2025 that may be closed by this pull request

Refactor to make use of QuestDB instead of tick csv files #1

Closed

mccaffers self-assigned this Nov 14, 2025

mccaffers added the enhancement New feature or request label Nov 14, 2025

mccaffers added this to @mccaffers Backtesting Project Tracking Nov 14, 2025

Copilot started reviewing on behalf of mccaffers November 14, 2025 18:18 View session

Copilot finished reviewing on behalf of mccaffers November 14, 2025 18:19

update build cache

6fca046

mccaffers moved this to In review in @mccaffers Backtesting Project Tracking Nov 14, 2025

Copilot AI reviewed Nov 14, 2025

View reviewed changes

mccaffers added 3 commits November 14, 2025 18:23

update build cache

182577d

fixing build config

b05c092

fixing build config

4a2e2f6

mccaffers requested a review from Copilot November 15, 2025 10:12

Copilot started reviewing on behalf of mccaffers November 15, 2025 10:12 View session

Copilot finished reviewing on behalf of mccaffers November 15, 2025 10:13

Copilot AI reviewed Nov 15, 2025

View reviewed changes

Updating CodeQL build and removing comments

ab03dad

mccaffers merged commit 626421e into main Nov 30, 2025
3 of 4 checks passed

github-project-automation bot moved this from In review to Done in @mccaffers Backtesting Project Tracking Nov 30, 2025

mccaffers deleted the 1-refactor-to-make-use-of-questdb-instead-of-tick-csv-files branch November 30, 2025 10:48

Provide feedback

Saved searches

Use saved searches to filter your results more quickly

Uh oh!

Updated codebase to use QuestDB instead of streaming files from S3 #2

Updated codebase to use QuestDB instead of streaming files from S3 #2

Uh oh!

mccaffers commented Nov 14, 2025

Copilot AI left a comment

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI Nov 14, 2025

Copilot AI left a comment

Copilot AI Nov 15, 2025

Copilot AI Nov 15, 2025

sonarqubecloud bot commented Nov 30, 2025

Uh oh!

Labels

2 participants

Updated codebase to use QuestDB instead of streaming files from S3 #2

Updated codebase to use QuestDB instead of streaming files from S3 #2

Uh oh!

Conversation

mccaffers commented Nov 14, 2025

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Key Changes

Reviewed Changes

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI Nov 14, 2025

Choose a reason for hiding this comment

Copilot AI left a comment

Choose a reason for hiding this comment

Pull Request Overview

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

Copilot AI Nov 15, 2025

Choose a reason for hiding this comment

sonarqubecloud bot commented Nov 30, 2025

Quality Gate failed

Uh oh!

Labels

2 participants