Deployment Notes

1. Deploying MarketData a) Stop the cron sevice. sudo service cron stop b) Check that MarketData is not running any tasks... ps xa | grep MarketData. mks might be running but that should be all b) Open Folder ARM64 and do dotnet clean followed by dotnet build c) Locate the build files in /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0 d) Backup the /opt/MarketData/MarketData files into /opt/MarketData/MarketData_Rollback e) Delete all but the logs folder from the /opt/MarketData/MarketData folder f) Copy files from /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0 into /opt/MarketData/MarketData g) Restart cron service.. sudo service cron restart h) Ensure that cron is running sudo service cron status

2. Deploying MarketDataService a) Disable the service sudo systemctl disable mks b) Stop the service. sudo systemctl stop mks This might take 30 seconds c) Copy all files from /opt/MarketData/MarketDataServer to /opt/MarketData/MarketDataServer_Rollback d) Delete all but the logs folder from /opt/MarketData/MarketDataServer e) Copy files from /home/pi/ARM64/MarketDataServer/bin/Debug/net8.0 into /opt/MarketData/MarketDataServer f) Enable the service sudo systemctl enable mks g) Start the service sudo systemctl restart mks h) Ensure that the service running systemctl status mks

3. Deploying IPMonitor a) Stop the cron service. sudo service cron stop b) Check that IPMonitor is not running. ps xa | grep ipmonitor notes: crontab -e shows the process in lower case c) Delete all but the ipaddress and logs folders from /opt/MarketData/IPMonitor d) Copy files from /home/pi/ARM64/IPMonitor/bin/Debug/net8.0 into /opt/MarketData/IPMonitor e) Restart the cron service. sudo service cron restart f) Ensure that cron cron is running sudo service cron status

End Of Month Processing Notes

There are four models in total CM20191031.TXT, MG20180131.TXT, MGSH20250331.TXT, CMT20200817.TXT

1. CMT20200817.TXT - There is no special end of month processing for this model. simply run ./runcmtrend.sh {DATE} as you normally would after the price capture. Repeat this excercise on Windows from the SAFERUN folder.

2. CM20191031.TXT - a) Start up the CNN Server on windows. C:\boneyard\CNN\Models\run_host.bat b) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells. c) After prices have been captured run the ./runcmmomentumeom.sh script with no arguments. d) Log on to Merrill Edge and confirm that any BUYS are available on Merrill Edge. If not then you will need to exclude any symbols that cannot be traded by adding them to the model file. e) After confirming the availability of purchases proceed to execute BUYS and SELLS on Merrill Edge. f) Reconcile all on the following day and book the executions into the system g) Repeat this excercise on Windows from the DEBUG folder after copying the model file(s) from the SAFERUN folder.

3. MG20180131.TXT - a) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells. b) After prices have been captured run the ./runmomentumeom.sh script with no arguments. c) Log on to Robinhood and confirm that any BUYS are available. If not then you will need to exclude any symbols that cannot be traded by adding them to the model file. d) After confirming the availability of purchases proceed to execute BUYS and SELLS. e) Reconcile all on the following day and book the executions into the system f) Repeat this excercise on Windows from the DEBUG folder after copying the model file(s) from the SAFERUN folder.

4. MGSH20250331.TXT - a) There is no associated Excel trade file for this model. b) After prices have been captured run the ./runmgshmomentum.sh MONTHLY {endofmonthdate} The date is not used during the month end processing but is validated in the script so just supply it. Repeat this excercise on Windows from the SAFERUN folder. Be sure to run ./runmgshmomentum MONTHLY {endofmonthdate} from the SAFERUN folder. c) Log on to Robinhood and confirm that any BUYS are available. If not then you will need to exclude any symbols that cannot be traded by adding them to the model file. d) After confirming the availability of purchases proceed to execute BUYS and SELLS. e) Reconcile all on the following day and book the executions into the system

   You will run the models on both Windows and Debian to compare the results. When running on Windows first copy all of the model files from SafeRun folder to both the Debug folder and also the Session folder. Commit any changes to GIT.

Installing MarketDataService service

1. change into the following folder /etc/systemd/system
2. create the mks.service file sudo nano mks.service
3. Place the following into the file

[Unit] Description=Market Data Server Service After=network-online.target Wants=network-online.target

[Service] User=pi WorkingDirectory=/opt/MarketData/MarketDataServer ExecStart=/opt/MarketData/MarketDataServer/mks Restart=always RestartSec=5 KillMode=process Environment=DOTNET_ROOT=/opt/dotnet StandardOutput=null

[Install] WantedBy=multi-user.target

4. Enable the service Enable the service: sudo systemctl enable mks.service. Start the service: sudo systemctl start mks.service. Check the status: sudo systemctl status mks.service Disable the service: sudo systemctl disable mks.service

Crontab for Production (Dia)

Here is the official latest crontab. Adding the monthly jobs to run on the 17th Notice that the monthly jobs will all run on the 15th of the month at 00:00 (zero hour and zero minute=midnight) CalcSticker will also start on the 15th but 15 seconds past midnight. It will wait for the completion of the specified processes

crontab is... (minute) (hour) (day of month) (month) (day of week)

The following is the production crontab for DIA. you can directly copy these lines into the cron schedule...

1. Stop the cron service. sudo service cron stop
2. Copy/Paste daya ... sudo crontab -e and then copy the below lines
3. Restart cron sudo service cron restart
4. Ensure cron is running sudo service cron status

sudo service cron reload , sudo service cron restart

cron helper app here -> https://cron.help/#/5_9-17__*_1-5

m h dom mon dow command

sudo service cron reload , sudo service cron restart

cron helper app here -> https://cron.help/#/5_9-17__*_1-5

DOTNET_ROOT=/opt/dotnet CRON_DIR=/opt/MarketData/MarketData CRON_DIR_IPMONITOR=/opt/MarketData/IPMonitor */15 6-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADHEADLINESWATCHLIST /WATCHLIST:Valuations > /dev/null 2>&1 */5 5-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADPREMARKETDATA > /dev/null 2>&1 */5 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEINTRADAYPRICING > /dev/null 2>&1 */30 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATELATESTANALYSTRATINGS > /dev/null 2>&1 0 19 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEDAILY2 /DATE:$(date -d"-0 days" +%m-%d-%Y) > /dev/null 2>&1 0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFINANCIALSTATEMENTS > /dev/null 2>&1 0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFUNDAMENTALS > /dev/null 2>&1 0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEHISTORICAL > /dev/null 2>&1 15 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk CALCSTICKER /WAITFORCOMPLETION:UPDATEFINANCIALSTATEMENTS,UPDATEFUNDAMENTALS,UPDATEHISTORICAL > /dev/null 2>&1 0 18 * * Sun cd $CRON_DIR ; /opt/MarketData/MarketData/mk OPTIMIZEDB > /dev/null 2>&1 0 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor IPMONITOR /FORCE:true > /dev/null 2>&1 10,20,30,40,50 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor > /dev/null 2>&1

Crontab for Adrastea (MariaDb Database Server)

The following is the production crontab for Adrastea to backup the database every saturday at noon. you can directly copy these lines into the cron schedule...

1. Stop the cron service. sudo service cron stop
2. Copy/Paste daya ... sudo crontab -e and then copy the below lines
3. Restart cron sudo service cron restart
4. Ensure cron is running sudo service cron status

sudo service cron reload , sudo service cron restart

cron helper app here -> https://cron.help/#/5_9-17__*_1-5

m h dom mon dow command

sudo service cron reload , sudo service cron restart

cron helper app here -> https://cron.help/#/5_9-17__*_1-5

CRON_DIR=/home/pi/ARM64/Scripts 0 12 * * 6 cd $CRON_DIR ; ./dump.sh > /dev/null 2>&1

Isonoe Backup Database Server, mks.service and MarketData cron jobs

There is a backup MariaDb database server running on Isonoe. The database is located on a USB drive on Isonoe at /mnt/mariadb There is a dump.sh script on Adrastea /home/pi/Boneyard/Import that can be used to dump the database There is also a send.sh script on Adrastea under /home/pi/Boneyard that can send the compressed backupdb.tar-gz file over to Isonoe There is a receive.sh script on Isonoe under /mnt/mariadb that will receive the backupdb.tar.gz file from Adrastea.

1. First run the dump.sh script on Adrastea (/home/pi/Boneyard/Import/dump.sh)
2. Next run the receive.sh script on Isonoe (/mnt/mariadb/receive.sh)
3. Next run the send.sh script on Adrastea (/home/pi/Boneyard/Import/send.sh)
4. Finally, run the restoredbzip.sh script on Isonoe (/mnt/mariadb/restoredbzip.sh)

Notes: There is a c++ program that performs the file transfer between Isonoe and Adrastea. The program is maintained on the Gitea repo under CPP/sstp The build project for this is maintained on Isonoe under /home/pi/CPP/sstp with libraries located under /home/pi/CPP/common

Isonoe also serves as a backup server for MarketData and IPMonitor. The crontab is installed and commented out. If you need to switch over to Isonoe ensure you grab the latest build files from the Gitea repository build for ARM64 and go to the Actions tab. Grab the latest build artifacts from the latest build action and deploy them to their respective /opt/MarketData folders. Here is the link to the actions. http://www.diversified-software.com:3000/Sean/ARM64/actions

-----------------------------------------------------------------------------------------

IPMonitor Notes

1. I added a network status verifier to the IPMonitor which will reboot the machine if the network has been down for 5 minutes

Miscellaneous Notes

To run unit tests we need to enable "Code Lens" so that the "Run Test" "Debug Test" options are visible and clickable in the source file. So make sure to go to File->Preferences->Settings in VsCode and type "Editor: Code Lense" and sure it's checked. If so you should see options to debug a single unit test right under the [TestMethod] annotation. To run unit tests from the CLI go to the unit test project folder in Terminal and run dotnet clean, dotnet build, dotnet test

To get a count of open files on raspbian lsof | wc -l

Git Delete Remote Branch git push <remote_name> --delete <branch_name> git push origin --delete <branch_name> git branch -r git branch -l

This removes any remote tracking branches that exist on local repository that are not in the remote git fetch origin --prune

You can use the following commands to update the list of local branches from remote: git fetch --prune

---

Adding notes on MariaDb as I plan to migrate the MySql over to one of the Raspberry PI 4 B's https://hevodata.com/learn/mysql-to-mariadb/

Notes on HttpClient https://josef.codes/you-are-probably-still-using-httpclient-wrong-and-it-is-destabilizing-your-software/

---

ARM version of marketdata libraries. sudo truncate syslog -- size 0 more on syslog and journal https://ubuntuforums.org/showthread.php?t=2495218 https://stackoverflow.com/questions/16995425/how-does-cmd-dev-null-21-work

journalctl --disk-usage # See log file disk use sudo journalctl --vacuum-size=200M # Drop log file size to 200M, if possible. sudo journalctl --vacuum-time=10d # Drop logs, over 10 days old

DEV NULL
ping a.b.c > /dev/null 2>&1

CRON SCHEDULE sudo crontab -e

** THE LATEST AND GREATEST IS ALWAYS AT THE BOTTOM

Also note that 1>/dev/null is synonymous to, but more explicit than >/dev/null

Financial

1. Economic Indicators - https://docs.tradingeconomics.com/

DOTNET related notes

Slow builds and C# Dev Kit Errors ps -u $USER -o pid,cmd | grep -E 'csdevkit|visualstudio-projectsystem-buildhost|vstest.console' | grep dotnet | awk '#{print $1}' | xargs -r kill -9 || true

This shows where dotnet performance is dotnet build /clp:PerformanceSummary