
# Deployment Notes
1) Deploying MarketData
  a) Stop the cron sevice. sudo service cron stop
  b) Check that MarketData is not running any tasks... ps xa | grep MarketData.  mks might be running but that should be all
  b) Open Folder ARM64 and do dotnet clean followed by dotnet build
  c) Locate the build files in /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0 
  d) Backup the /opt/MarketData/MarketData files into /opt/MarketData/MarketData_Rollback
  e) Delete all but the logs folder from the /opt/MarketData/MarketData folder
  f) Copy files from /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0 into /opt/MarketData/MarketData
  g) Restart cron service..  sudo service cron restart
  h) Ensure that cron is running sudo service cron status

2) Deploying MarketDataService
  a) Disable the service  sudo systemctl disable mks
  b) Stop the service.  sudo systemctl stop mks  This might take 30 seconds
  c) Copy all files from /opt/MarketData/MarketDataServer to /opt/MarketData/MarketDataServer_Rollback
  d) Delete all but the logs folder from /opt/MarketData/MarketDataServer
  e) Copy files from /home/pi/ARM64/MarketDataServer/bin/Debug/net8.0 into /opt/MarketData/MarketDataServer
  f) Enable the service sudo systemctl enable mks
  g) Start the service sudo systemctl restart mks
  h) Ensure that the service running systemctl status mks


3) Deploying IPMonitor
   a) Stop the cron service.  sudo service cron stop
   b) Check that IPMonitor is not running.  ps xa | grep ipmonitor    notes: crontab -e shows the process in lower case
   c) Delete all but the ipaddress and logs folders from /opt/MarketData/IPMonitor
   d) Copy files from /home/pi/ARM64/IPMonitor/bin/Debug/net8.0 into /opt/MarketData/IPMonitor
   e) Restart the cron service.  sudo service cron restart
   f) Ensure that cron cron is running sudo service cron status

# End Of Month Processing Notes
There are four models in total CM20191031.TXT, MG20180131.TXT, MGSH20250331.TXT, CMT20200817.TXT

1) CMT20200817.TXT - There is no special end of month processing for this model.  simply run ./runcmtrend.sh {DATE} as
   you normally would after the price capture.

2) CM20191031.TXT - 
   a) Start up the CNN Server on windows.  C:\boneyard\CNN\Models\run_host.bat
   b) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells.
   c) After prices have been captured run the ./runcmmomentumeom.sh script with no arguments.
   d) Log on to Merrill Edge and confirm that any BUYS are available on Merrill Edge.  If not then you will need to exclude any 
      symbols that cannot be traded by adding them to the model file.
   e) After confirming the availability of purchases proceed to execute BUYS and SELLS on Merrill Edge.
   f) Reconcile all on the following day and book the executions into the system

3) MG20180131.TXT - 
   a) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells.
   b) After prices have been captured run the ./runmomentumeom.sh script with no arguments.
   c) Log on to Robinhood and confirm that any BUYS are available.  If not then you will need to exclude any 
      symbols that cannot be traded by adding them to the model file.
   d) After confirming the availability of purchases proceed to execute BUYS and SELLS.
   e) Reconcile all on the following day and book the executions into the system

4) MGSH20250331.TXT - 
   a) There is no associated Excel trade file for this model.
   b) After prices have been captured run the runmgshmomentum.sh MONTHLY {endofmonthdate}
      The date is not used during the month end processing but is validated in the script so just supply it.
      Repeat this excercise on Windows.  Be sure to run ./runmgshmomentum MONTHLY {endofmonthdate} from the SAFERUN folder.
   c) Log on to Robinhood and confirm that any BUYS are available.  If not then you will need to exclude any 
      symbols that cannot be traded by adding them to the model file.
   d) After confirming the availability of purchases proceed to execute BUYS and SELLS.
   e) Reconcile all on the following day and book the executions into the system    

   You will run the models on both Windows and Debian to compare the results.
   When running on Windows first copy all of the model files from SafeRun folder to both the Debug folder and also the Session
   folder.  Commit any changes to GIT.

# Installing MarketDataService service
1) change into the following folder /etc/systemd/system
2) create the mks.service file  sudo nano mks.service
3) Place the following into the file

[Unit]
Description=Market Data Server Service
After=network-online.target
Wants=network-online.target

[Service]
User=pi  # Or the user your executable runs as
WorkingDirectory=/opt/MarketData/MarketDataServer
ExecStart=/opt/MarketData/MarketDataServer/mks
Restart=always
RestartSec=5  # Seconds to wait before restarting (optional)
KillMode=process # Ensures the service is killed gracefully
Environment=DOTNET_ROOT=/opt/dotnet
StandardOutput=null

[Install]
WantedBy=multi-user.target

4) Enable the service 
Enable the service: sudo systemctl enable mks.service. 
Start the service: sudo systemctl start mks.service. 
Check the status: sudo systemctl status mks.service
Disable the service: sudo systemctl disable mks.service


# Crontab for Production (Dia)
Here is the official latest crontab.  Adding the monthly jobs to run on the 17th
Notice that the monthly jobs will all run on the 15th of the month at 00:00  (zero hour and zero minute=midnight)
CalcSticker will also start on the 15th but 15 seconds past midnight.  It will wait for the completion of the specified processes

crontab is...
(minute) (hour) (day of month) (month) (day of week)


The following is the production crontab for DIA.  you can directly copy these lines into the cron schedule...
1) Stop the cron service.  sudo service cron stop
2) Copy/Paste daya ...  sudo crontab -e     and then copy the below lines
3) Restart cron sudo service cron restart
4) Ensure cron is running sudo service cron status

# sudo service cron reload , sudo service cron restart
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
# m h  dom mon dow   command
# sudo service cron reload , sudo service cron restart
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
DOTNET_ROOT=/opt/dotnet
CRON_DIR=/opt/MarketData/MarketData
CRON_DIR_IPMONITOR=/opt/MarketData/IPMonitor
*/15 6-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADHEADLINESWATCHLIST /WATCHLIST:Valuations > /dev/null 2>&1
*/5 5-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADPREMARKETDATA > /dev/null 2>&1
*/5 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEINTRADAYPRICING > /dev/null 2>&1
*/30 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATELATESTANALYSTRATINGS > /dev/null 2>&1
0 19 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEDAILY2 /DATE:$(date -d"-0 days" +\%m-\%d-\%Y) > /dev/null 2>&1
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFINANCIALSTATEMENTS > /dev/null 2>&1
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFUNDAMENTALS > /dev/null 2>&1
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEHISTORICAL > /dev/null 2>&1
15 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk CALCSTICKER /WAITFORCOMPLETION:UPDATEFINANCIALSTATEMENTS,UPDATEFUNDAMENTALS,UPDATEHISTORICAL > /dev/null 2>&1
0 18 * * Sun cd $CRON_DIR ; /opt/MarketData/MarketData/mk OPTIMIZEDB > /dev/null 2>&1
0 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor IPMONITOR /FORCE:true > /dev/null 2>&1 
10,20,30,40,50 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor > /dev/null 2>&1


# Crontab for Adrastea (MariaDb Database Server)

The following is the production crontab for Adrastea to backup the database every saturday at noon.  you can directly copy these lines into the cron schedule...
1) Stop the cron service.  sudo service cron stop
2) Copy/Paste daya ...  sudo crontab -e     and then copy the below lines
3) Restart cron sudo service cron restart
4) Ensure cron is running sudo service cron status
# sudo service cron reload , sudo service cron restart
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
# m h  dom mon dow   command
# sudo service cron reload , sudo service cron restart
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
CRON_DIR=/home/pi/ARM64/Scripts
0 12 * * 6 cd $CRON_DIR ; ./dump.sh > /dev/null 2>&1

# Isonoe Backup Database Server
There is a backup MariaDb database server running on Isonoe.
The database is located on a USB drive on Isonoe at /mnt/mariadb
There is a dump.sh script on Adrastea /home/pi/Boneyard/Import that can be used to dump the database 
There is also a send.sh script on Adrastea under /home/pi/Boneyard that can send the compressed backupdb.tar-gz file over to Isonoe
There is a receive.sh script on Isonoe under /mnt/mariadb that will receive the backupdb.tar.gz file from Adrastea.
1) First run the dump.sh script on Adrastea (/home/pi/Boneyard/Import/dump.sh)
2) Next run the receive.sh script on Isonoe (/mnt/mariadb/receive.sh)
3) Next run the send.sh script on Adrastea (/home/pi/Boneyard/Import/send.sh)
4) Finally, run the restoredbzip.sh script on Isonoe (/mnt/mariadb/restoredbzip.sh)

Notes: There is a c++ program that performs the file transfer between Isonoe and Adrastea.
       The program is maintained on the Gitea repo under CPP/sstp
       The build project for this is maintained on Isonoe under /home/pi/CPP/sstp with libraries located under /home/pi/CPP/common

# IPMonitor Notes
1) I added a network status verifier to the IPMonitor which will reboot the machine if the network has been down for 5 minutes


# Miscellaneous Notes
TO-DO
1) MariaDb online
2) MarketDataServer 
  dotnet add package Microsoft.AspNet.WebApi.SelfHost --version 5.3.0
  dotnet add package System.ServiceModel.Primitives  --version 4.0.0

Git Delete Remote Branch
git push <remote_name> --delete <branch_name>
git push origin --delete <branch_name>
git branch -r
git branch -l

You can use the following commands to update the list of local branches from remote:
git fetch --prune

******************************************************************************************************
Adding notes on MariaDb as I plan to migrate the MySql over to one of the Raspberry PI 4 B's 
https://hevodata.com/learn/mysql-to-mariadb/

Notes on HttpClient
https://josef.codes/you-are-probably-still-using-httpclient-wrong-and-it-is-destabilizing-your-software/


*******************************************************************************************************

ARM version of marketdata libraries.
sudo truncate syslog -- size 0
more on syslog and journal
https://ubuntuforums.org/showthread.php?t=2495218
https://stackoverflow.com/questions/16995425/how-does-cmd-dev-null-21-work

journalctl --disk-usage   # See log file disk use
  sudo journalctl --vacuum-size=200M    # Drop log file size to 200M, if possible.
  sudo journalctl --vacuum-time=10d     # Drop logs, over 10 days old
  
DEV NULL  
ping a.b.c > /dev/null 2>&1 

  
CRON SCHEDULE
sudo crontab -e

** THE LATEST AND GREATEST IS ALWAYS AT THE BOTTOM

Also note that 1>/dev/null is synonymous to, but more explicit than >/dev/null
