252 lines
13 KiB
Markdown
252 lines
13 KiB
Markdown
# Deployment Notes
|
|
1) Deploying MarketData
|
|
a) Stop the cron sevice. sudo service cron stop
|
|
b) Check that MarketData is not running any tasks... ps xa | grep MarketData. mks might be running but that should be all
|
|
b) Open Folder ARM64 and do dotnet clean followed by dotnet build
|
|
c) Locate the build files in /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0
|
|
d) Backup the /opt/MarketData/MarketData files into /opt/MarketData/MarketData_Rollback
|
|
e) Delete all but the logs folder from the /opt/MarketData/MarketData folder
|
|
f) Copy files from /home/pi/ARM64/MarketData/MarketData/bin/Debug/net8.0 into /opt/MarketData/MarketData
|
|
g) Restart cron service.. sudo service cron restart
|
|
h) Ensure that cron is running sudo service cron status
|
|
|
|
2) Deploying MarketDataService
|
|
a) Disable the service sudo systemctl disable mks
|
|
b) Stop the service. sudo systemctl stop mks This might take 30 seconds
|
|
c) Copy all files from /opt/MarketData/MarketDataServer to /opt/MarketData/MarketDataServer_Rollback
|
|
d) Delete all but the logs folder from /opt/MarketData/MarketDataServer
|
|
e) Copy files from /home/pi/ARM64/MarketDataServer/bin/Debug/net8.0 into /opt/MarketData/MarketDataServer
|
|
f) Enable the service sudo systemctl enable mks
|
|
g) Start the service sudo systemctl restart mks
|
|
h) Ensure that the service running systemctl status mks
|
|
|
|
|
|
3) Deploying IPMonitor
|
|
a) Stop the cron service. sudo service cron stop
|
|
b) Check that IPMonitor is not running. ps xa | grep ipmonitor notes: crontab -e shows the process in lower case
|
|
c) Delete all but the ipaddress and logs folders from /opt/MarketData/IPMonitor
|
|
d) Copy files from /home/pi/ARM64/IPMonitor/bin/Debug/net8.0 into /opt/MarketData/IPMonitor
|
|
e) Restart the cron service. sudo service cron restart
|
|
f) Ensure that cron cron is running sudo service cron status
|
|
|
|
# End Of Month Processing Notes
|
|
There are four models in total CM20191031.TXT, MG20180131.TXT, MGSH20250331.TXT, CMT20200817.TXT
|
|
|
|
1) CMT20200817.TXT - There is no special end of month processing for this model. simply run ./runcmtrend.sh {DATE} as
|
|
you normally would after the price capture. Repeat this excercise on Windows from the SAFERUN folder.
|
|
|
|
2) CM20191031.TXT -
|
|
a) Start up the CNN Server on windows. C:\boneyard\CNN\Models\run_host.bat
|
|
b) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells.
|
|
c) After prices have been captured run the ./runcmmomentumeom.sh script with no arguments.
|
|
d) Log on to Merrill Edge and confirm that any BUYS are available on Merrill Edge. If not then you will need to exclude any
|
|
symbols that cannot be traded by adding them to the model file.
|
|
e) After confirming the availability of purchases proceed to execute BUYS and SELLS on Merrill Edge.
|
|
f) Reconcile all on the following day and book the executions into the system
|
|
g) Repeat this excercise on Windows from the DEBUG folder after copying the model file(s) from the SAFERUN folder.
|
|
|
|
3) MG20180131.TXT -
|
|
a) Prepare the usual Excel trade file (C:\boneyard\marketdata\TradeBook) so that you can track the buys and sells.
|
|
b) After prices have been captured run the ./runmomentumeom.sh script with no arguments.
|
|
c) Log on to Robinhood and confirm that any BUYS are available. If not then you will need to exclude any
|
|
symbols that cannot be traded by adding them to the model file.
|
|
d) After confirming the availability of purchases proceed to execute BUYS and SELLS.
|
|
e) Reconcile all on the following day and book the executions into the system
|
|
f) Repeat this excercise on Windows from the DEBUG folder after copying the model file(s) from the SAFERUN folder.
|
|
|
|
4) MGSH20250331.TXT -
|
|
a) There is no associated Excel trade file for this model.
|
|
b) After prices have been captured run the ./runmgshmomentum.sh MONTHLY {endofmonthdate}
|
|
The date is not used during the month end processing but is validated in the script so just supply it.
|
|
Repeat this excercise on Windows from the SAFERUN folder. Be sure to run ./runmgshmomentum MONTHLY {endofmonthdate} from the SAFERUN folder.
|
|
c) Log on to Robinhood and confirm that any BUYS are available. If not then you will need to exclude any
|
|
symbols that cannot be traded by adding them to the model file.
|
|
d) After confirming the availability of purchases proceed to execute BUYS and SELLS.
|
|
e) Reconcile all on the following day and book the executions into the system
|
|
|
|
You will run the models on both Windows and Debian to compare the results.
|
|
When running on Windows first copy all of the model files from SafeRun folder to both the Debug folder and also the Session
|
|
folder. Commit any changes to GIT.
|
|
|
|
# Installing MarketDataService service
|
|
1) change into the following folder /etc/systemd/system
|
|
2) create the mks.service file sudo nano mks.service
|
|
3) Place the following into the file
|
|
|
|
[Unit]
|
|
Description=Market Data Server Service
|
|
After=network-online.target
|
|
Wants=network-online.target
|
|
|
|
[Service]
|
|
User=pi
|
|
WorkingDirectory=/opt/MarketData/MarketDataServer
|
|
ExecStart=/opt/MarketData/MarketDataServer/mks
|
|
Restart=always
|
|
RestartSec=5
|
|
KillMode=process
|
|
Environment=DOTNET_ROOT=/opt/dotnet
|
|
StandardOutput=null
|
|
|
|
[Install]
|
|
WantedBy=multi-user.target
|
|
|
|
4) Enable the service
|
|
Enable the service: sudo systemctl enable mks.service.
|
|
Start the service: sudo systemctl start mks.service.
|
|
Check the status: sudo systemctl status mks.service
|
|
Disable the service: sudo systemctl disable mks.service
|
|
|
|
|
|
# Crontab for Production (Dia)
|
|
Here is the official latest crontab. Adding the monthly jobs to run on the 17th
|
|
Notice that the monthly jobs will all run on the 15th of the month at 00:00 (zero hour and zero minute=midnight)
|
|
CalcSticker will also start on the 15th but 15 seconds past midnight. It will wait for the completion of the specified processes
|
|
|
|
crontab is...
|
|
(minute) (hour) (day of month) (month) (day of week)
|
|
|
|
|
|
The following is the production crontab for DIA. you can directly copy these lines into the cron schedule...
|
|
1) Stop the cron service. sudo service cron stop
|
|
2) Copy/Paste daya ... sudo crontab -e and then copy the below lines
|
|
3) Restart cron sudo service cron restart
|
|
4) Ensure cron is running sudo service cron status
|
|
|
|
# sudo service cron reload , sudo service cron restart
|
|
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
|
|
# m h dom mon dow command
|
|
# sudo service cron reload , sudo service cron restart
|
|
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
|
|
DOTNET_ROOT=/opt/dotnet
|
|
CRON_DIR=/opt/MarketData/MarketData
|
|
CRON_DIR_IPMONITOR=/opt/MarketData/IPMonitor
|
|
*/15 6-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADHEADLINESWATCHLIST /WATCHLIST:Valuations > /dev/null 2>&1
|
|
*/5 5-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk LOADPREMARKETDATA > /dev/null 2>&1
|
|
*/5 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEINTRADAYPRICING > /dev/null 2>&1
|
|
*/30 9-16 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATELATESTANALYSTRATINGS > /dev/null 2>&1
|
|
0 19 * * 1-5 cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEDAILY2 /DATE:$(date -d"-0 days" +\%m-\%d-\%Y) > /dev/null 2>&1
|
|
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFINANCIALSTATEMENTS > /dev/null 2>&1
|
|
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEFUNDAMENTALS > /dev/null 2>&1
|
|
0 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk UPDATEHISTORICAL > /dev/null 2>&1
|
|
15 0 15 * * cd $CRON_DIR ; /opt/MarketData/MarketData/mk CALCSTICKER /WAITFORCOMPLETION:UPDATEFINANCIALSTATEMENTS,UPDATEFUNDAMENTALS,UPDATEHISTORICAL > /dev/null 2>&1
|
|
0 18 * * Sun cd $CRON_DIR ; /opt/MarketData/MarketData/mk OPTIMIZEDB > /dev/null 2>&1
|
|
0 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor IPMONITOR /FORCE:true > /dev/null 2>&1
|
|
10,20,30,40,50 * * * * cd $CRON_DIR_IPMONITOR ; /opt/MarketData/IPMonitor/ipmonitor > /dev/null 2>&1
|
|
|
|
|
|
# Crontab for Adrastea (MariaDb Database Server)
|
|
|
|
The following is the production crontab for Adrastea to backup the database every saturday at noon. you can directly copy these lines into the cron schedule...
|
|
1) Stop the cron service. sudo service cron stop
|
|
2) Copy/Paste daya ... sudo crontab -e and then copy the below lines
|
|
3) Restart cron sudo service cron restart
|
|
4) Ensure cron is running sudo service cron status
|
|
# sudo service cron reload , sudo service cron restart
|
|
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
|
|
# m h dom mon dow command
|
|
# sudo service cron reload , sudo service cron restart
|
|
# cron helper app here -> https://cron.help/#*/5_9-17_*_*_1-5
|
|
CRON_DIR=/home/pi/ARM64/Scripts
|
|
0 12 * * 6 cd $CRON_DIR ; ./dump.sh > /dev/null 2>&1
|
|
|
|
# Isonoe Backup Database Server, mks.service and MarketData cron jobs
|
|
There is a backup MariaDb database server running on Isonoe.
|
|
The database is located on a USB drive on Isonoe at /mnt/mariadb
|
|
There is a dump.sh script on Adrastea /home/pi/Boneyard/Import that can be used to dump the database
|
|
There is also a send.sh script on Adrastea under /home/pi/Boneyard that can send the compressed backupdb.tar-gz file over to Isonoe
|
|
There is a receive.sh script on Isonoe under /mnt/mariadb that will receive the backupdb.tar.gz file from Adrastea.
|
|
1) First run the dump.sh script on Adrastea (/home/pi/Boneyard/Import/dump.sh)
|
|
2) Next run the receive.sh script on Isonoe (/mnt/mariadb/receive.sh)
|
|
3) Next run the send.sh script on Adrastea (/home/pi/Boneyard/Import/send.sh)
|
|
4) Finally, run the restoredbzip.sh script on Isonoe (/mnt/mariadb/restoredbzip.sh)
|
|
|
|
Notes: There is a c++ program that performs the file transfer between Isonoe and Adrastea.
|
|
The program is maintained on the Gitea repo under CPP/sstp
|
|
The build project for this is maintained on Isonoe under /home/pi/CPP/sstp with libraries located under /home/pi/CPP/common
|
|
|
|
Isonoe also serves as a backup server for MarketData and IPMonitor. The crontab is installed and commented out.
|
|
If you need to switch over to Isonoe ensure you grab the latest build files from the Gitea repository build for ARM64 and go to the Actions tab.
|
|
Grab the latest build artifacts from the latest build action and deploy them to their respective /opt/MarketData folders.
|
|
Here is the link to the actions. http://www.diversified-software.com:3000/Sean/ARM64/actions
|
|
|
|
# -----------------------------------------------------------------------------------------
|
|
|
|
# IPMonitor Notes
|
|
1) I added a network status verifier to the IPMonitor which will reboot the machine if the network has been down for 5 minutes
|
|
|
|
|
|
# Miscellaneous Notes
|
|
To run unit tests we need to enable "Code Lens" so that the "Run Test" "Debug Test" options are visible and clickable in the source file.
|
|
So make sure to go to File->Preferences->Settings in VsCode and type "Editor: Code Lense" and sure it's checked.
|
|
If so you should see options to debug a single unit test right under the [TestMethod] annotation.
|
|
To run unit tests from the CLI go to the unit test project folder in Terminal and run dotnet clean, dotnet build, dotnet test
|
|
|
|
To get a count of open files on raspbian lsof | wc -l
|
|
|
|
|
|
|
|
|
|
Git Delete Remote Branch
|
|
git push <remote_name> --delete <branch_name>
|
|
git push origin --delete <branch_name>
|
|
git branch -r
|
|
git branch -l
|
|
|
|
This removes any remote tracking branches that exist on local repository that are not in the remote
|
|
git fetch origin --prune
|
|
|
|
|
|
|
|
You can use the following commands to update the list of local branches from remote:
|
|
git fetch --prune
|
|
|
|
******************************************************************************************************
|
|
Adding notes on MariaDb as I plan to migrate the MySql over to one of the Raspberry PI 4 B's
|
|
https://hevodata.com/learn/mysql-to-mariadb/
|
|
|
|
Notes on HttpClient
|
|
https://josef.codes/you-are-probably-still-using-httpclient-wrong-and-it-is-destabilizing-your-software/
|
|
|
|
|
|
*******************************************************************************************************
|
|
|
|
ARM version of marketdata libraries.
|
|
sudo truncate syslog -- size 0
|
|
more on syslog and journal
|
|
https://ubuntuforums.org/showthread.php?t=2495218
|
|
https://stackoverflow.com/questions/16995425/how-does-cmd-dev-null-21-work
|
|
|
|
journalctl --disk-usage # See log file disk use
|
|
sudo journalctl --vacuum-size=200M # Drop log file size to 200M, if possible.
|
|
sudo journalctl --vacuum-time=10d # Drop logs, over 10 days old
|
|
|
|
DEV NULL
|
|
ping a.b.c > /dev/null 2>&1
|
|
|
|
|
|
CRON SCHEDULE
|
|
sudo crontab -e
|
|
|
|
** THE LATEST AND GREATEST IS ALWAYS AT THE BOTTOM
|
|
|
|
Also note that 1>/dev/null is synonymous to, but more explicit than >/dev/null
|
|
|
|
# Financial
|
|
1) Economic Indicators - https://docs.tradingeconomics.com/
|
|
|
|
|
|
# DOTNET related notes
|
|
Slow builds and C# Dev Kit Errors
|
|
ps -u $USER -o pid,cmd | grep -E 'csdevkit|visualstudio-projectsystem-buildhost|vstest.console' | grep dotnet | awk '#{print $1}' | xargs -r kill -9 || true
|
|
|
|
Kill MSBuild that don't close when exiting vscode
|
|
pkill -f MSBuild.dll
|
|
ps aux | grep MSBuild.dll | grep -v grep | awk '{print $2}' | xargs sudo kill -9
|
|
|
|
This shows where dotnet performance is
|
|
dotnet build /clp:PerformanceSummary
|
|
|
|
# kill vscode
|
|
sudo killall code
|