Market Simulation (Part 07): Sockets (I)

Introduction

In the previous article, Market Simulation (Part 06): Transferring Information from MetaTrader 5 to Excel, I explained how, using simple means and pure MQL5, you can transfer quote data from MetaTrader 5 into Excel. Of course, that was only a very basic example, whose purpose was to be as didactic as possible. However, you may have noticed that the quote updates were not happening in real time. Still, the knowledge presented in the previous article aims to demonstrate something else that we will actually need to do.

But since there are several ways to achieve the same thing, and what really matters to us is always the result, I wanted to show that yes, there is a simple way to transfer data from MetaTrader 5 into other programs, such as Excel. However, the main idea is not to transfer data from MetaTrader 5 to Excel, but the opposite, that is, to transfer data from Excel or any other program to MetaTrader 5.

There are several ways to accomplish this. Some are simpler, others more elaborate. Some can do this in real time, while others introduce a slight delay. But what matters is: Can we control MetaTrader 5, or some type of application running within it, using external data? This kind of question may seem trivial, but it opens up important possibilities. For example, you could conduct some sort of study or analysis using an external program specifically designed for such tasks. You could define points of interest in terms of price or quotation and place such data somewhere where an application (an Expert Advisor, Service, Script, or even an Indicator) can use it. This can then provide us with signals directly on the chart or, in the case of an Expert Advisor, assist us in decision-making.

At this point, only your imagination and your level of interest will limit what can be done and how far you can go. But before we move on to developing something we actually need for the Replay/Simulator, I want to show something else. For that, these concepts, ideas, and possibilities need to be presented so that you can truly understand what will be developed and implemented later and thus make the best possible use of all this knowledge.

What I will show in this article is a step forward on the same topic covered previously. Because, as I said, the method demonstrated earlier does not allow us to send data in real time. Although that is already sufficient for what we need in the replay/simulator, it does not meet our needs if we want information in real time.

Understanding the Concept

As I mentioned in the previous article, using RTD or DDE does not guarantee bidirectional communication between MetaTrader 5 and any external program. But the same concept used in RTD or DDE can be applied in a way that gives us what we want: bidirectional communication. To accomplish this, we will need to do something a little different from what we have done so far. Throughout this development phase, I have kept everything in pure MQL5. However, there are cases where pure MQL5 is not truly sufficient—not because of what it can do, but because of what external programs cannot do without assistance. The issue COMES down to one thing: SOCKETS.

Sockets are deeply embedded in computing. If you do not know what a socket is or what it is used for, I suggest that you look it up, especially if you intend to work with certain types of solutions. Using sockets gives us a tremendous degree of freedom. I will not go deep into the topic here, as it is extensive and full of possibilities requiring careful study if you really want to work with sockets. But I will offer a brief explanation so you, dear reader, do not feel completely lost, so you understand what we will be doing and what is happening.

TCP Protocol

The first thing you need to understand is that there are countless different types of sockets. Each serves a different purpose. However, the most common are Stream and DataGram sockets. The stream type, also known as a flow socket, is commonly used in reliable bidirectional communications. This word reliable relates to the fact that it is always tested to ensure information is received. Stream sockets use the Transmission Control Protocol, commonly known as TCP.

This type of socket has another characteristic: it is designed so that data packets always arrive in a specific sequence. In other words, you can transfer information as if it were an unfolding list, since packets arrive in order. This makes it clear where it is most commonly used: for example, in remote application control, where losing data during transmission is unacceptable and we need guarantees that information will be received. However, this makes the protocol somewhat slower in practice. Typically, the structure is as follows:

But in terms of programming, what matters is something else. It's about how this protocol is selected and implemented in the code we will create. There are small structural differences in the code depending on whether TCP is used in Windows or Linux. Although such differences exist, they do not affect the communication itself. That is, a Windows application can communicate with a Linux application via TCP, and both will understand each other perfectly. Provided the communication rules are followed. The same applies to the next protocol, which we will discuss in the following section.

UDP Protocol

Having covered the Stream type, let us now understand the DataGram type. A DataGram socket is completely different from a Stream/TCP socket. This is because the Datagram socket is intended for unidirectional, unreliable communication. But what does that mean? How can communication be unidirectional and unreliable? Let us break it down.

First, this socket uses the protocol known as UDP. This protocol is not a real connection, but merely a method of sending data. To help you understand, imagine writing a letter and sending it. Once you send it, you do not know if or when it will actually reach its destination.

But things get more complicated. After sending the letter, you realize you forgot some details. So you write another letter and send it. This can happen multiple times - each letter is sent separately.

Here's the key detail: if the postal service works correctly and none of the letters are lost, the recipient may receive them all at once, or in the order you sent them, or in a completely random order.

Think about the work you, as the recipient, would have to do to understand the message. And if you were the sender expecting a response, you would have no guarantee that the recipient actually got or understood the letters. For this reason, the UDP protocol is considered unidirectional and unreliable. But despite how it sounds, it is still extremely useful.

There are situations, such as data probing, where we do not need the data to be ordered, nor do we need all of it. We can afford to lose some, yet still have enough to understand the information.

A typical example is video or voice calls, where speed is the priority. Another application would be remote probes or sensors, where the goal is simply to confirm a certain condition. In such cases, confirmation of receipt is unnecessary as long as enough data arrives for analysis. This protocol typically follows the structure shown below:

Hopefully this has already sparked your curiosity, prompting you to explore the subject more deeply. Indeed, it is extremely extensive and interesting. Here in the MQL5 community, you will find numerous references to the topic, both in articles and forum discussions. In any case, the topic is vast. This brief introduction is only meant to help you understand what we will do next.

Programming a Modest System

Here we need to open a parenthesis. To demonstrate what will be done, we must use very simple code. Since MQL5 does not allow us to create a server directly within the language, we will use MQL5 to create the client side. Yes, we are going to build a Client–Server model. The server side will be written using external programming. In this case, you may create it using any language or platform that supports network servers. But there are important details you must pay attention to, such as the port and the type of protocol. To simplify things as much as possible, I will use code that you can study directly from the source. In other words, we will use publicly available code from the web. The server code can be found on the Microsoft website, accessible via this link: Servidor WinSock.

At the end of the article, I will leave this reference again so you can study the code step by step, along with many other details related to sockets. This applies not only to the server but also to the client, which is also explained on the same page.

IMPORTANT NOTE: The server I am referencing is a very simple model. It receives one connection, captures what is sent to it, and immediately closes the connection, being unable to receive another. It returns the exact data sent by the client. In other words, it is an echo server used only for demonstration.

However, we will do something slightly different for the client. We will create the client directly in MQL5, skipping the C/C++ client code available on the site. This will allow you to understand how communication actually works. Keep in mind that you could even turn MetaTrader 5 into a server, but that would require using a DLL. I do not want to introduce this type of resource at this moment. At least not yet.

So let us get started. You may use the code I indicated, or create your own in any language or platform. The idea here is simply to build a modest system, one that is as simple as possible but serves as a trigger for you to understand how things work. Once the server is compiled and running, we can move on to the MQL5 portion. For that, open MetaEditor and create the client in pure MQL5.

Running the MetaTrader Side

Before we establish a connection with our local server, we need to understand how to actually make a connection. MetaTrader 5, by default, prevents you from connecting to any server other than the trading server. This is a security measure that you must accept and understand. It is not appropriate for a platform like MetaTrader to connect freely to arbitrary addresses. Therefore, a small step must be taken before MetaTrader will allow such a connection.

To explain this in a very simple way, we will use something found directly in the MQL5 documentation. So the code below is not mine but is from the official documentation and can be found under SocketCreate. The source code is shown below exactly as it appears in the documentation:

001. //+------------------------------------------------------------------+
002. //|                                                SocketExample.mq5 |
003. //|                        Copyright 2018, MetaQuotes Software Corp. |
004. //|                                             https://www.mql5.com |
005. //+------------------------------------------------------------------+
006. #property copyright   "Copyright 2018, MetaQuotes Software Corp."
007. #property link        "https://www.mql5.com"
008. #property version     "1.00"
009. #property description "Add Address to the list of allowed ones in the terminal settings to let the example work"
010. #property script_show_inputs
011.  
012. input string Address="www.mql5.com";
013. input int    Port   =80;
014. bool         ExtTLS =false;
015. //+------------------------------------------------------------------+
016. //| Send command to the server                                       |
017. //+------------------------------------------------------------------+
018. bool HTTPSend(int socket,string request)
019.   {
020.    char req[];
021.    int  len=StringToCharArray(request,req)-1;
022.    if(len<0)
023.       return(false);
024. //--- if secure TLS connection is used via the port 443
025.    if(ExtTLS)
026.       return(SocketTlsSend(socket,req,len)==len);
027. //--- if standard TCP connection is used
028.    return(SocketSend(socket,req,len)==len);
029.   }
030. //+------------------------------------------------------------------+
031. //| Read server response                                             |
032. //+------------------------------------------------------------------+
033. bool HTTPRecv(int socket,uint timeout)
034.   {
035.    char   rsp[];
036.    string result;
037.    uint   timeout_check=GetTickCount()+timeout;
038. //--- read data from sockets till they are still present but not longer than timeout
039.    do
040.      {
041.       uint len=SocketIsReadable(socket);
042.       if(len)
043.         {
044.          int rsp_len;
045.          //--- various reading commands depending on whether the connection is secure or not
046.          if(ExtTLS)
047.             rsp_len=SocketTlsRead(socket,rsp,len);
048.          else
049.             rsp_len=SocketRead(socket,rsp,len,timeout);
050.          //--- analyze the response
051.          if(rsp_len>0)
052.            {
053.             result+=CharArrayToString(rsp,0,rsp_len);
054.             //--- print only the response header
055.             int header_end=StringFind(result,"\r\n\r\n");
056.             if(header_end>0)
057.               {
058.                Print("HTTP answer header received:");
059.                Print(StringSubstr(result,0,header_end));
060.                return(true);
061.               }
062.            }
063.         }
064.      }
065.    while(GetTickCount()<timeout_check && !IsStopped());
066.    return(false);
067.   }
068. //+------------------------------------------------------------------+
069. //| Script program start function                                    |
070. //+------------------------------------------------------------------+
071. void OnStart()
072.   {
073.    int socket=SocketCreate();
074. //--- check the handle
075.    if(socket!=INVALID_HANDLE)
076.      {
077.       //--- connect if all is well
078.       if(SocketConnect(socket,Address,Port,1000))
079.         {
080.          Print("Established connection to ",Address,":",Port);
081.  
082.          string   subject,issuer,serial,thumbprint;
083.          datetime expiration;
084.          //--- if connection is secured by the certificate, display its data
085.          if(SocketTlsCertificate(socket,subject,issuer,serial,thumbprint,expiration))
086.            {
087.             Print("TLS certificate:");
088.             Print("   Owner:  ",subject);
089.             Print("   Issuer:  ",issuer);
090.             Print("   Number:     ",serial);
091.             Print("   Print: ",thumbprint);
092.             Print("   Expiration: ",expiration);
093.             ExtTLS=true;
094.            }
095.          //--- send GET request to the server
096.          if(HTTPSend(socket,"GET / HTTP/1.1\r\nHost: www.mql5.com\r\nUser-Agent: MT5\r\n\r\n"))
097.            {
098.             Print("GET request sent");
099.             //--- read the response
100.             if(!HTTPRecv(socket,1000))
101.                Print("Failed to get a response, error ",GetLastError());
102.            }
103.          else
104.             Print("Failed to send GET request, error ",GetLastError());
105.         }
106.       else
107.         {
108.          Print("Connection to ",Address,":",Port," failed, error ",GetLastError());
109.         }
110.       //--- close a socket after using
111.       SocketClose(socket);
112.      }
113.    else
114.       Print("Failed to create a socket, error ",GetLastError());
115.   }
116. //+------------------------------------------------------------------+

Source code from the MQL5 documentation

This code is perfect for explaining how to work with and adjust connections in MetaTrader 5. Because in addition to being very simple, it gives us the possibility of remote access. In other words, we will actually access a webpage. I believe that many readers with some experience in network programming will understand this code without difficulty. But for those who know nothing about the subject, I will give a quick explanation, because we will later need to understand some of the concepts shown here.

This code is a script and must be compiled as such. It is not possible to use sockets inside indicators. Read the documentation for more details. Let's now see how it works. On line 12, we specify the name of the website to be accessed. On line 13, we indicate the port that will be used. Specifically, a port that responds to HTTP requests. In other words, a web server. See the documentation for what to expect from each type of port.

Well, when the code starts (on line 71) the first thing we do is create a socket (line 73). Up to this point, everything is quite simple. However, we must verify that the created socket is valid, which is done on line 75. On line 78, we attempt to connect to the server and port defined on lines 12 and 13. If this fails, you will see something similar to the image below:

This is likely caused by MetaTrader 5 blocking the connection. To enable the connection, press CTRL + O and add an exception to the rule, as shown below:

Once you make the change shown above, you will be able to run the script again, and the result will look similar to the image below:

But how did this happen? How were we able to retrieve information from the web server? Wouldn't a WebRequest call be necessary for this? Actually, not necessarily. The WebRequest call does almost exactly what we just did, but at a slightly higher level. Because WebRequest creates an abstraction around the socket underneath. Therefore, the result is the same in both approaches. We explored the WebRequest function in past articles. If you want more details, see:

• Developing a trading Expert Advisor from scratch (Part 15): Accessing data on the web (II)
• Developing a trading Expert Advisor from scratch (Part 16): Accessing data on the web (II)
• Developing a trading Expert Advisor from scratch (Part 17): Accessing data on the web (III)

What was done in those three articles can also be done using sockets directly. However, you will need to write a bit more code to achieve the same result. But let's return to our system in this article. After compiling the server code referenced in the previous section, you must leave it running before executing the script I will show shortly. Remember that you can create your own server in any language you want. The only requirement for now is that the data received by the server be sent back to the client, so that we get the behavior I will explain.

So let's look at the MQL5 code for our client. You can see it in full below.

01. //+------------------------------------------------------------------+
02. #property copyright "Daniel Jose"
03. #property version   "1.00"
04. #property script_show_inputs 
05. //+------------------------------------------------------------------+ 
06. input string user00 = "127.0.0.1";          //Address
07. input int    user01 = 27015;                //Port
08. //+------------------------------------------------------------------+
09. bool ConnectionWrite(int socket, string szMsg, bool TLS = false)
10. {
11.     char buff[];
12.     int  len = StringToCharArray(szMsg, buff) - 1;
13. 
14.     if (len < 0) return false;
15.     Print("To Server:", szMsg);
16.     return (TLS ? SocketTlsSend(socket, buff, len) : SocketSend(socket, buff, len)) == len;
17. }
18. //+------------------------------------------------------------------+
19. string ConnectionRead(int socket, uint timeout)
20. {
21.     uint len;
22.     int  ret;
23.     char resp[];
24.     string info;
25.     uint timeout_check = GetTickCount() + timeout;
26.     
27.     info = "";
28.     do
29.     {
30.             len = SocketIsReadable(socket);
31.             ret = SocketRead(socket, resp, len, timeout);
32.             if (ret > 0)
33.                     info += CharArrayToString(resp, 0, ret);
34.     }while ((GetTickCount() < timeout_check) && (!_StopFlag));
35.     
36.     return info;
37. }
38. //+------------------------------------------------------------------+
39. void OnStart()
40. {
41.     int socket = SocketCreate();
42.     
43.     if (socket == INVALID_HANDLE)
44.     {
45.             Print("Unable to create socket. Error: ", GetLastError());
46.             return;
47.     }
48.     if (!SocketConnect(socket, user00, user01, 1000))
49.     {
50.             Print("Connection with the address [", user00,"] in port: ",user01, " failed. Error code: ", GetLastError());
51.             SocketClose(socket);
52.             return;
53.     }
54.     
55.     Print("Trying to send ", ConnectionWrite(socket, "Testing Server Echo...") ? "Ok..." : "Failed...");
56.     
57.     Print("From Server:", ConnectionRead(socket, 1000));
58. 
59.     SocketClose(socket);
60. }
61. //+------------------------------------------------------------------+

Demonstration code

Let's understand a few details in this code before seeing how it runs in MetaTrader 5. The first important point is line six. This line tells us where our server is located. In this example, we specify that it is on the same machine as MetaTrader 5. Line seven specifies the server port. This is very important: this port must be the same as the one defined in the server code. That is why I referenced publicly available code that is clearly explained on the website. If in doubt, study the documentation at that site. I will include it again as a reference at the end of this article.

Basically, all functions in this MQL5 script are self-explanatory. And if you have questions, simply read the documentation to understand the details of what is happening. However, there is one part where things are not so clear and deserve some explanation.

Notice that on line 28 we enter a loop. Its purpose is to read what the server posts to the socket. But why use a loop? Wouldn't it be enough to simply read the socket once? This may seem odd, but there is a reason for it. When something is sent to the socket, we do not know how it will be sent. We only know that it will be sent. The information may arrive in a single packet or in several small packets, even when the message is relatively short.

This is why many people new to socket programming experience difficulty. They assume the information will arrive in a single block. But it may actually be divided into many blocks or packets. Can we know this in advance? In reality, there is no simple way to know. All we can expect is that the sent data will be received when using TCP, as explained earlier. But the information will usually be posted and received in packet form.

To better understand, look at line 55. Here we call line nine, which returns the information to be placed into the socket. On line 15, we define the message. On line 16, we send it. How the server will receive this message, and how many packets it will be split into, is no longer under our control - the system itself handles this.

Then, on line 57, we read what the server has published. Pay attention to this: we are not reading what we sent, but what the server published. Because the server we are using is an echo server, the same message we sent will be returned to us. This is essentially like performing a PING on the server.

But just as we do not control how the message is transmitted, we also do not know how the server will return it in terms of packet count. So, if you remove the loop on line 28 and execute only the call on line 31, and then try to convert the server response to a string using line 33, you will likely notice that the message may appear fragmented as if some part were missing. However, at other times, it will arrive complete, because the message is short. And it is precisely this kind of behavior that many people do not understand.

I know this may seem confusing and even inefficient. It may seem like it serves no purpose. But I want to emphasize something: if you understand how these two codes work - both the server and the one executed in MetaTrader 5 - you will realize we can achieve a great deal.

But before running this script, you must inform MetaTrader 5 that the address used should be allowed. Therefore, the same value indicated in line six must appear in the list of allowed addresses in MetaTrader 5. You should see something like the image below:

With this modification in place, we can run both programs. To show how to do this, especially if you lack experience running multiple processes, I have included the video below demonstrating how to test both programs locally. I want to emphasize that the purpose of the video is not to promote any specific tool, but simply to show those unfamiliar with the process how to run programs in multiple work threads and thus enable development without major difficulties.

Final Thoughts

In this article, I provided a brief introduction to one of the most complex concepts you can use in MetaTrader 5: sockets. The purpose of this article was to explain the bare basics of how to use sockets within MetaTrader 5 so that you can at least understand some necessary and involved concepts. Although the content here is very basic, it is essential for anyone wishing to study sockets and their applications.

But because all we did here was send an echo to the server, it may not be entirely clear how much we can truly do. To help with that, in the next article we will build something more elaborate. This socket concept will significantly improve the quality of your replay/simulator system, because we still need to implement additional features, such as the order system. And I am seriously considering building it in a more advanced way. But for you, dear reader, to truly understand what will be programmed, you must understand these small concepts I am presenting here.

If there is one thing that will make you study deeply when it comes to programming, it is this: sockets. When the subject is sockets, you never know enough. And when you think you have explored everything about them, you will inevitably discover that there is still more to learn. So do not miss the next article: I will show something very interesting and simple to implement on this topic.

Link

Microsoft WinSock documentation

List of port numbers