Good morning,
As a Data Scientist, I am very interested in the world of AI (Artificial Intelligence). Especially today, in this world where almost everything is connected to the Internet and computers, which means, all is controlled by a set of programs. Welcome to the world of AI!
There are plenty of resources on the internet on topics like this. Just type in Google the keywords you want and then you will find many websites.
This article is divided into 4 sections:
Part #1: Context and architecture of ML.
Part #2: Output - Process - Input
Part #3: Coding.
Part #4: Results.
Let's start.
What is "Machine Learning"?
Machine learning is a technology that allows a computer system to learn from existing data and perform specific tasks based on that learning. ML is designed to be self-contained and able to learn on its own without requiring explicit direction from the user. This learning ability is analogous to the functioning of the human brain.
Machine learning is based on several disciplines, such as statistics, mathematics, computer science, and data mining, which allow machines to learn by analyzing data without the need to reprogram or modify them. give specific instructions. ML works can vary depending on the approach used, but it generally follows common learning principles and methods.
The fundamental steps of how machine learning works include:
- Data collection: Gather the data needed to train the machine learning model.
- Data mining: Analyzing and understanding data to extract useful insights and identify trends.
- Model selection: Choose the type of machine learning model that best suits the task at hand.
- Model training and testing: Train the model with the available data, then test it to assess its performance.
- Performance evaluation of ML results: Measure the efficiency and accuracy of the model based on the results obtained.
Machine Learning is widely used in many fields such as speech recognition, fraud detection, product recommendation, autonomous driving, etc. It has become an essential tool for effectively exploiting data and making decisions based on in-depth analyses.
Machine learning, often abbreviated as "ML", is a technology that can study existing data and perform certain tasks based on what it learns. ML technology is a machine that we can develop to be able to learn by itself without direction from the user. Just like the human brain. It has the ability to learn.
And this time, I want to share my experience from my Machine Learning development project. Machine learning can be applied in many disciplines. One of the experiences that I had is in the medical and financial world. Like the detection of Covid-19 and the detection of credit card fraud. But in this project, I invite you to learn to develop Machine Learning which can predict price actions, and it is very possible that it can also be applied to Forex and Crypto. Are you ready?
But it should be noted, this article is for study purposes only. As for applying to the world of Real Trading, do so at your own risk.
OK... it's starting to get serious now. This project is a challenge or a challenge for me. Since the Covid-19 pandemic hit the world and we all experienced periods of lockdown, I see the positive side of taking advantage of these moments to learn how to trade. And I have been learning since then to make a trading robot or an Expert Advisor.
Objective
What is this Machine Learning for? To be honest, I have a dream that this Machine Learning could become an assistant and inspire novice Traders, like me… :D or those of you who might need it. And it is also possible to inspire traders all over the planet... :). This is the vision and mission of this project.
I usually learn to trade using the Metatrader application, especially Metatrader4. Why? The reason is, KISS......Keep It Simple, Stupid...... Moreover, Metatrader is also available for cross-platforms. Additionally, metatrader4 with MQL4 programming language is also widely available on the internet.
Problems
The problems I and (probably) most newbie traders face are:
- When should I take a BUY or SELL position?
- At what price should I BUY or SELL?
- When should I ENTER or EXIT the market?
Solution
The solution is...lots of choices. As the term "All roads lead to Rome", means many solutions or ways to achieve our targets or goals. As long as we focus on one of the goals we set for ourselves. And be consistent.
Some of the solutions I've found on the internet, from easiest to hardest, are as follows:
- Buy a trading robot, such as an EA. Prices vary. From free to millions of dollars.
- Join a copy trading program. All you have to do is choose experienced brokers and expert traders and then copy them.
- Look for trading signals. Many websites or apps provide trading signals. All you have to do is install it, then wait for the notification whenever there is a signal, then trade.
- Take a trading class or course.
- Learn to trade on your own. To be more independent.
Each of the above solutions has its advantages and disadvantages. I experienced the #1, #2 and #3. While for the #4, I took online courses on Youtube, but in the end, I was not satisfied. Because I am CONFIDENT that I have abilities and skills, and I believe I can do it BETTER. Therefore, I decided to take solution #5. Another reason for me, the solution #5 is the right solution and in line with my vision and mission. Moreover, we can also use this trading skill anywhere, anytime, without limit. Is it true or right ? :D
In simple words, the architecture of the system or the project that will be built can be seen in the diagram below.
MT4 <-----JSON----->ML
That's it... simple, isn't it? Why is architecture like that? Metatrader4 (MT4) uses MQL4 programming language, similar to C/C++ language, but the library is not sufficient for machine learning to perform calculations. MQL4 is suitable to make EA (Expert Advisor) or Indicator. Meanwhile, on the other hand, we have the Python programming language with a bunch of libraries that can be used to build machine learning. Well, because MT4 and Python do not have a library to communicate and exchange data. To connect between MT4 and ML Python, JSON is used for data transmission. MT4 will send historical data in the form of OHLC (Open, High, Low, Close) prices from the number of candles/bars we specify. We will use this OHLC data as input data for machine learning. Then the price prediction output from machine learning will be sent back to MT4 using JSON format.
In this article, I want to share the technical challenges of a Machine Learning project. Where are we going to start? Do you prefer a forward or backward method? For ease of understanding, I will explain using the backward method, which is from output - process - input.
Output
Pair : EURUSD - TF : 30
Bullish / Bearish : Bearish
Prev Close : 1.10053 at : 2023-04-28 10:00:00
-------------------------------------------------------------------------------------------
Model NextClose Closest Action MinPred MaxPred Score Accuracy(%)
-------------------------------------------------------------------------------------------
1. RF 1.10457 1.10457 BUY 1.10457 1.10552 0.00000 99.96
2. LSTM 1.10254 1.10159 BUY 1.10159 1.10254 0.00000 99.94
3. Fine-LSTM 1.10266 1.10187 BUY 1.10187 1.10266 0.00000 99.94
4. SARIMAX 1.09671 1.09763 SELL 1.09561 1.64402 0.00003 99.48
5. Hybrid 1.09904 1.09922 SELL 1.09830 1.42732 NaN NaN
6. Future-LSTM 1.10420 1.10074 BUY 1.10074 1.10420 NaN NaN
-------------------------------------------------------------------------------------------
***The Closest Pred : 1.10074 >>> BUY
The Nearest Next Close : 1.09904 >>> STRONG SELL
The Lowest Pred : 1.09561
The Highest Pred : 1.64402
The highest Accuracy : 99.96 %
Predicting is DONE! -- Pair: EURUSD at TF: 30
This is an example of the output from Machine Learning. There are some informations about the pair used, in this example we used forex, EURUSD, with TimeFrame (TF) 30 Minutes (M30). There is information about market conditions (Bearish/Bullish), Close Price on the previous TF, namely 'Prev_Close', complete with the datetime format. Next, the table below gives the prediction results of 6 models where each model displays the prediction results for 'NextClose', 'Closest' (the price closest to Prev_Close), Action, Minimum Prediction, Max_Prediction, Score, and Accuracy (%). In addition, just below the table is the summary of the results of the 6 models. The 'Closest' is the number closest to a set of prediction results for each model to the Prev_Close number. Meanwhile, 'Nearest Next Close' is the number closest to the NextClose prediction.
What if we add a Metatrader chart display like Figure 1 below?
The red line (dots) is the result of Machine Learning. Next N-bars price movement prediction was received by Metatrader.
So… what now? With the predictions from Machine Learning and the visualization, will you dare to take trade positions? If you don't dare enough, that's okay… Indeed, an important factor in the world of trading is the mindset or mentality. :D
Well....what do you think if you get information like the above one? Does this information help you to make a decision?
OK… that was the Output. Now I want to explain the process.
Process
To make it easier to understand, I will provide some sort of server-side and client-side pseudocode.
Server Side:
1. Create Socket Server at [IP_Address] and [Port]
2. While True: a. Receive JSON_Data from MT4 (client)
3. Parsing JSON_Data
4. Create a new dataframe
5. Adding Features and Indicators into dataframe
6. Define Target
7. Splitting data into Data_Train, Data_Valid, Data_Test
8. Training Phase
a. Train with Random Forest
b. Train with LSTM
c. Train with Finetuned-LSTM
d. Train with SARIMAX
e. Train with Hybrid (LSTM + SARIMAX)
9. Validation Phase
10. Testing Phase
11. Evaluation Phase
12. Predicting Phase
13. Future Prediction
14. Candle Pattern Detection
a. Detecting Bearish candle
b. Detecting Bullish candle
15. Analyzing Results
16. Send the results via JSON to MT4 (client)
Client Side:
1. Connect to Server via socket at [IP_Address] and [Port]
2. Create JSON_Data
3. If Socket.Send and Object_Click = 'Connect & Train': Sending JSON_Data to Server
4. While Socket.Receive: Parsing JSON_Data
5. While Object_Click = 'Predict' Draw objects
6. While Object_Click = 'Place Order' Open Trade
7. While Object_Click = 'Re-Train' Sending JSON_Data to Server
8. Read JSON_Data: Open Trade Automatically (Optional)
Input - Raw data for machine learning
The next section is the input. As I said before, the input data is in the form of OHLC (Open, High, Low, Close) prices from a number of candles/bars taken from Metatrader (see Figure 2 below ). For this, we need to create an Expert Advisor (EA) using MQL4 language and run it on MetaTrader. The pseudocode can be seen in the client-side section. So, the client side uses the MQL4 programming language, while the server side uses Python. Between MQL4 and Python exchanging information or data via JSON. Metatrader provides the input in the form of OHLC prices to be processed by Machine Learning and the output in the form of prediction results that are sent back to Metatrader.
Coding - [Learn] Programming
We start by creating a Class Socket Server.
For each model that we will develop, we will each create a procedure. The goal is to ease the debugging process and further development. So, for example, in this project, we will use 6 models. Then we will develop, for each model, a function. As an example, here is the Training function using the Random Forest model.
In this project, we are going to create several regressor model functions. Why? Well, to know the performance of a model, of course, we need several models for comparison. So that we can determine which model has the best performance and prediction accuracy. Regarding other regressor models, such as LSTM and SARIMAX, as well as Hybrid, please expand your own.... :D :D
I already gave an example with the Random Forest regressor model.
The next function that can be considered an important feature of this project is the bearish/bullish candlestick detector. So, here is the candlestick pattern detector function.
OK, that was part of Server-side. We are now learning to create programs for the client-side. This time we will use the MQL4 language for Metatrader4. And if we install Metatrader4, we have an editor to create MQL4 programs, namely MetaEditor.
As you can see in the MQL4 client-side script above, we are copying the OHLC (Open-High-Low-Close) prices data and packaging it into a JSON format, then sending it to the server via socket.
Results
In this article, I will not provide backtest results, like EA in general. Why? Because ML is not an EA. And ML cannot be backtested. But ML can show its performance from the training and test results. So, this ML can be tested or forwardtest, either on a demo account or a real live account of Metatrader4.
Future works
Future work that can be developed in this project are as follows:
- Integration of economic news. So, later in the decision-making analysis for an action BUY/SELL, Machine Learning will collect from the internet all economic news by using scraping and NLP methods.
- Reinforcement learning. We could implement reinforcement learning such as DQN or DDQN Agent.
- Create a server application (web-based). Currently, ML is only accessible on a localhost machine. It takes additional steps to be accessible via the Internet. Of course, security is an important aspect at this stage.
- Create mobile applications. With the mobile app, I hope that everyone who uses this app can make a prediction anytime, anywhere.
So, this is only a brief description of learning how to program using Python and MQL4 in this project.
I'm sure that you still have a lot of unclear things in this project. Well, that's because of my limitations in providing details. But please leave a trace below in the comment column or contact me directly.
Colmar, July 7th, 2023