Routing Protocol for Heterogeneous Wireless Mesh Networks

The introduction of heterogeneous wireless mesh technologies provides an opportunity for higher network capacity, wider coverage, and higher quality of service (QoS). Each wireless device utilizes different standards, data formats, protocols, and access technologies. However, the diversity and complexity of such technologies create challenges for traditional control and management systems. This paper proposes a heterogeneous metropolitan area network architecture that combines an IEEE 802.11 wireless mesh network (WMN) with a long-term evolution (LTE) network. In addition, a new heterogeneous routing protocol and a routing algorithm based on reinforcement learning called cognitive heterogeneous routing are proposed to select the appropriate transmission technology based on parameters from each network. The proposed heterogeneous network overcomes the problems of sending packets over long paths, island nodes, and interference in WMNs and increases the overall capacity of the combined network by utilizing unlicensed frequency bands instead of buying more license frequency bands for LTE. The work is validated through extensive simulations that indicate that the proposed heterogeneous WMN outperforms the LTE and Wi-Fi networks when used individually. The simulation results show that the proposed network achieves an increase of up to 200% in throughput compared with Wi-Fi-only networks or LTE-only networks.


I. INTRODUCTION
I NTERNET traffic is expected to increase three to five times over the next three years due to the growing number of connected mobile devices. The number of connected devices and machine-to-machine communications is expected to exceed the number of the population by a factor of 2 over the next three years. It is predicted that within the next decade, a more advanced Internet infrastructure will be required to support this increase in Internet traffic [1], [2].
Next-generation wireless networks need to address several challenges, including the cost to cover high-density areas, crowded events, or large areas, or respond to temporary changes in demand, e.g., large sporting events. The cost estimation is dependent on the number of required base stations and the cost to rent frequency bands. Interoperability is another challenge as many devices use different operating systems, protocols, and access technologies. Network reliability is also an important issue that needs to be addressed to ensure that systems are able to tolerate faults in case of disasters [3].
The internetworking of different wireless technologies, particularly long-term evolution (LTE) and wireless local area networks, is one of the key opportunities for developing nextgeneration wireless networks [3]. LTE is an evolution of the third-generation standard, which provides wide coverage and a peak transmission rate ranging from 100 to 326.4 Mb/s on the downlink (from base station to user equipment) and from 50 to 86.4 Mb/s on the uplink (from user equipment to base station) depending on the antenna configuration and modulation depth. Due to the advanced technologies employed in LTE networks, it is used by major mobile operators around the world to cope with the high traffic demands. However, LTE networks employ licensed frequency bands, and therefore, to provide more bandwidth, an additional cost is introduced to either buy more frequency bands (which may not be available in all regions) or invest in a higher density of base stations.
Another promising wireless architecture for the next generation of wireless networks is wireless mesh networks (WMNs). The WMN is a paradigm developed to provide wide network coverage without using centralized infrastructure [4]. Therefore, WMNs are a feasible choice to provide a backbone network for metropolitan area networks. In such networks, gateways (wireless nodes with high-speed wired connection to external Internet) are employed to provide Internet connection to the mesh network. This architecture offers cost-effective ubiquitous wireless connection to the Internet in large areas through multihop transmission to the gateway, and vice versa. However, the major drawbacks of using WMNs are their limitations in terms of capacity, system performance, and guaranteed wireless link quality. The causes of those problems originate in the multihop nature of the network. When data packets traverse more hops in a large WMN, they may either fail to reach their destination or consume too many network resources. Moreover, in case of link or node failure, some nodes may become isolated from the network due to the lack of a path to the destination or gateway and form what is known as an "island node." The use of heterogeneous technologies in wireless networks improves the overall network performance by distributing the load across different network technologies [5]- [7]. However, switching from one transmission device to another presents challenges as each wireless device utilizes different standards and data formats. Furthermore, routing packets through a heterogeneous network requires a new mechanism to exchange control messages among the different networks.
In this paper, a new heterogeneous WMN (HetMeshNet) architecture is proposed to overcome these drawbacks and provide wider coverage with better quality of service (QoS). LTE [8] and IEEE 802.11 [9] are employed in each wireless node to form the infrastructure for the proposed architecture. In addition, a new routing protocol is developed to enable the communication among the heterogeneous wireless nodes. A set of routing tables and control messages is employed to manage the interoperability of the proposed heterogeneous WMN. Moreover, this paper proposes a new heterogeneous routing algorithm called cognitive heterogeneous routing (CHR) that enables heterogeneous networks to work as a single network. The packets are routed through different wireless technologies based on the QoS metrics of each network. To optimize the selection of the transmission technology, a new reinforcement learning algorithm based on Q-learning is developed to allow each node to learn from previous actions and improve the network performance. Extensive simulations under different scenarios and traffic demand in the networks indicate a throughput gain with up to 200% in the proposed heterogeneous network when compared with LTE-and Wi-Fi-only networks.
This paper proposes a novel heterogeneous network architecture that overcomes the drawbacks of each transmission technology utilized in the network. The use of WMNs increases network capacity by utilizing unlicensed frequency bands that reduce the cost of buying more LTE-licensed frequency. The LTE network is utilized to avoid a low-quality Wi-Fi link or connect an island node when link failure occurs. To enable communication between WMN and LTE networks, a new routing protocol is developed that defines a set of routing tables that is necessary to allow packet exchange between the two different networks. Then, a cross-layer heterogeneous routing algorithm called CHR is proposed that employs QoS parameters from each node with both LTE and Wi-Fi transmission technologies to select the best radio access technology. This paper is organized as follows. Section II highlights related work on heterogeneous networks. Section III introduces the system architecture. Section IV describes the proposed heterogeneous routing protocol, which is then experimentally verified using simulation in Section V. Finally, Section VI offers concluding remarks and suggestions for future work.

II. RELATED WORK
This section reviews advanced approaches to managing heterogeneous wireless networks and discusses the advantages and disadvantages of each approach. The management of heterogeneous networks involves employing different technologies that cooperate with each other to improve both the system performance and the metrics utilized to select the transmission technology. The proposed work employs WMNs in heterogeneous network architecture, and therefore, this review first highlights related work on WMN routing protocols and metrics. This is followed by a review of heterogeneous network architectures.

A. Routing Protocols in WMNs
There are two types of routing protocols in WMNs. The first type consists of reactive routing protocols in which the route is created on demand by flooding the network with route requests. Route selection is maintained only for nodes that transmit traffic to a particular destination. Examples of this type of routing are ad hoc on-demand distance vectors [10] and dynamic source routing [11]. Reactive routing causes some delay due to the fact that a route is created only when there are data ready to be sent. The second type of routing protocol consists of proactive or table-driven routing protocols. They maintain a table of the entire destination in the network by periodically distributing an update of the routing table to all nodes. Destination-sequenced distance vector [12] and optimized link state routing (OLSR) [13] are examples of this type of routing protocol. The route table maintains the route for each destination; transmission begins with no delay if there are packets ready to be sent. However, some overhead is added for distributing routing table information among the nodes in the network. Some routing protocols, which are known as hybrid routing protocols, combine reactive and proactive routing to reduce the overhead of route discovery by employing proactive routing for near nodes and generating routes for far nodes using on-demand routing [14].

B. Routing Metrics in WMNs
The most widely utilized metrics in WMN routing protocols select the shortest path to the gateway based on the hop count, i.e., the number of nodes between the source and the destination. However, prior research has recognized the shortcoming of hop count metrics in WMNs because the shortest path metric results in a congested path [15]. Therefore, many researchers employ quality-aware metrics, which dynamically evaluate link quality characteristics to improve network performance. Some of these metrics employ link loss ratio to select the path to the gateway. One of the most widely cited measures is expected transmission count (ETX) [16], which estimates the required number of transmissions for successful data delivery between two nodes. ETX does not consider bandwidth, packet size, or link interference; therefore, the metric does not perform well with a network that has a high transmission rate and a large packet size. Expected transmission time [17] enhances ETX by considering the packet size and the link bandwidth in calculating the metric. However, this metric does not consider the load and link interference. Interference and channel switching (MIC) is proposed [18] as an alternative metric to ETT. MIC is topology dependent and selects paths with a minimum number of nodes that share the wireless channel. However, MIC fails to indicate whether the interferer node has data to transmit, as the interferer cannot cause interference when there is no transmission.
Another key link characteristic is the transmission rate. IEEE 802.11 supports multiple transmission rates; for each rate, there is a different transmission range and a different interference range. The simplest and most widely adopted algorithms in controlling the transmission rates are based on gathering the statistics of unsuccessful transmissions on the sender side to estimate the interference. The earliest algorithm in this category is the automatic rate fallback (ARF) [19], which sets a threshold based on the number of successful and failed transmissions to increase and decrease the transmission rate. Several enhancements to the ARF have been proposed to avoid unnecessary updates in transmission rates. For example, adaptive ARF [20] changes the threshold of switching the transmission rates adaptively. MadWifi ONOE [21] enhances ARF by assigning credits to each transmission rate based on the loss rates and then selects the transmission rates with the highest credits. These rate adaptation algorithms are developed for infrastructurebased wireless networks and not for WMNs, and therefore, they do not consider the competing nodes accessing shared channels. The recently developed rate adaptation algorithm based on reinforcement learning (RARE) [22] sets the transmission rate based on the link quality of the neighboring nodes and the load on the Wi-Fi device. Thus, the transmission rate estimates the amount of interference and collision with other nodes and the load on the node. The best link quality is that which provides a higher transmission rate. This algorithm is developed for WMNs and designed to work in highly congested multihop networks.

C. Heterogeneous Network Architectures
This part of the review discusses architectures that utilize different types of transmission technologies in wireless networks. The heterogeneous network architectures are reviewed according to the type of transmission technologies employed in the heterogeneous networks (i.e., cellular networks and Wi-Fi devices).
A cellular/multihop Wi-Fi architecture is proposed to relay data packets for clients that suffer from low channel quality or to offload a congested cell by forwarding the traffic from this cell to other noncongested cells [23]- [26]. These networks utilize the multihop Wi-Fi network as an auxiliary network to redirect traffic from one cell to another.
Heterogeneous networks split data among broadband and Wi-Fi wireless networks [5]- [7], [27]- [31] to increase network capacity. One approach is to distribute traffic among networks fairly [5] by employing load-balancing algorithms. Other architectures employ wireless characteristics to distribute data among networks. For instance, networks with better wall penetration are utilized for indoor communication, whereas networks with higher frequency bands are employed for outside communication [6], [7]. Traffic priority is employed to manage packet flow in heterogeneous networks [27] in which only sensitive packets from the Wi-Fi network are forwarded through the cellular network to avoid unreliable links. Wi-Fi access points are also utilized to create picocells to offload congested cells in cellular networks [28].
IEEE 802.11-based vehicular ad hoc networks (VANETs) and LTE networks are employed to form a hybrid network in which some nodes in VANETs are elected to work as a gateway to forward traffic to the LTE base stations [32], [33]. The access network is selected based on a set of QoS parameters to improve the network performance throughout the mobile path of vehicles.
Other recent research aims to improve cellular networks by employing a mixture of macrocells and small cells, such as microcells, picocells, and femtocells [34]- [39]. The use of small cells improves the frequency reuse by employing lower transmission power, which produces less interference and increases the data rate of cellular networks.
Heterogeneous wireless ad hoc networks are another type of heterogeneous networks that employ wireless nodes with different wireless capabilities and multiple radio devices to enhance the routing protocol [40]- [46]. This type of network employs heterogeneous communication devices (e.g., Bluetooth, 802.11), wireless devices with different transmission ranges, or different medium access protocols to improve ad hoc networks without considering the use of broadband networks, such as cellular, LTE, or WiMAX.
A promising approach is to equip cellular base stations with different wireless access technologies and frequency bands to reduce the interference among neighboring cells [47]. The coverage of each base station is divided into a number of regions based on the modulation and coding scheme utilized by each wireless technology in the base station.
The architectures reviewed focus on employing different wireless technologies either to offload the load on one network and use the network as an auxiliary network or to provide load balancing between these two networks. Both LTE and Wi-Fi networks are employed as two separate networks, and thus, the user device chooses one of the networks to forward its packets.
The heterogeneous network architecture proposed in this paper differs from the previously mentioned networks in that it employs a multihop Wi-Fi mesh network to provide a costeffective wide coverage area where only some of the nodes are connected to the Internet, and the LTE network is tightly deployed with the Wi-Fi network. The proposed network manages the heterogeneous radio access technologies as a part of a single virtual network. Thus, the proposed heterogeneous network utilizes the advantages of using WMN unlicensed free frequency bands, whereas the LTE network mitigates the issues that WMN suffers from such as interference and island nodes. In addition, the Wi-Fi network is utilized to improve the LTE network on the edge of the cell by propagating data to another node with better channel quality.
This paper contributes to the body of knowledge in this area of research by addressing a number of important issues. First, it proposes a new architecture for heterogeneous networks by combining WMN architecture with LTE architecture. The Internet gateway connecting the heterogeneous network to the Internet is a new feature of this architecture; the WMNs employ mesh gateways to connect the multihop network to the Internet through the Internet gateway. The second contribution is the heterogeneous routing protocol, which prescribes how the heterogeneous devices communicate with each other. The purpose of the proposed protocol is to create the required routing tables to allow the heterogeneous wireless devices to send packets between the two networks. The routing protocol specifies the set of routing tables that each node needs to maintain and the set of control messages that the heterogeneous nodes exchange among each other. It also specifies the type of transmission technology to be used to transmit these control messages. The third contribution of this work is creating a routing algorithm, which is called CHR, that defines the steps required to select the transmission device at the nodes that have both LTE and Wi-Fi devices. Reinforcement learning is employed to learn from the previous actions and optimize the network performance.

III. SYSTEM ARCHITECTURE
The proposed heterogeneous WMN (HetMeshNet) considers the coexistence of multiple wireless technologies as well as a wired network. It employs the following types of nodes: • heterogeneous nodes (HetNode)-nodes with both Wi-Fi and LTE capability; • mesh gateway nodes-nodes with Wi-Fi and wired connection; • LTE base stations-also known as evolved NodeB (eNodeB or eNB); • Internet gateway nodes-nodes that connect all the networks to the Internet using a high-speed wired network; and • client nodes-used by end users or sensors. Fig. 1 shows an example of the proposed HetMeshNet architecture. It comprises several types of network components. First, the LTE network consists of a number of cells distributed in the region. An LTE base station is located in each cell. Second, a number of HetNodes are deployed in the network, each of which is capable of utilizing multiple transmission technologies. The heterogeneous nodes (HetNodes) are equipped with Wi-Fi and LTE network interface cards. The mesh gateway nodes are the third type of nodes, which connect the WMN to the Internet gateway. The Internet gateway acts as a server; it provides Internet connection to both the LTE and WMN networks. Finally, the client nodes could be a human using a mobile phone, a laptop, or any other device connected to the Internet (e.g., a sensor sending data to the Internet).
Each heterogeneous node in this architecture can transmit its data to the Internet using either Wi-Fi or LTE. For example, if a HetNode sends the packet to a neighboring node via Wi-Fi, the neighboring node could forward the packet using LTE or Wi-Fi. Thus, both technologies are employed to mitigate the disadvantages that each technology could suffer from, including overloaded nodes or poor-quality wireless channels. By contrast, if a node is receiving packets from the Internet (downlink), the Internet gateway decides whether to forward data via LTE or WMN. Note that, in contrast to uplink, if Wi-Fi is selected for the downlink transmission, the intermediate nodes cannot switch back to LTE.
In this paper, an urban hotspot scenario is considered, in which a large number of devices wish to access the Internet simultaneously. No interference is assumed between the networks because different frequency bands are employed by the wireless networks. Each cell in the network employs the same architecture (as shown in Fig. 1); therefore, this paper is focused on a single cell of the LTE network.

IV. HETEROGENEOUS ROUTING PROTOCOL
This section provides a detailed description of the proposed routing protocol of the heterogeneous WMN. The new routing protocol proposed employs metrics from both networks to switch dynamically between transmission technologies. The proposed protocol consists of two main components: the heterogeneous routing tables and a routing algorithm. In a heterogeneous wireless network, the routing protocols need to employ metrics from all the technologies that might be utilized by a node.

A. Heterogeneous Routing Tables
Each type of node uses different transmission technologies, and each transmission technology employs a different network address. To route packets between these different networks, each type of node maintains a routing table to forward data packets from different networks just as if they were coming from the same network. First, the Internet gateway node needs a routing table to forward data packets to and from the Internet for both WMN and LTE networks. Second, each heterogeneous node maintains a table of routes to the other heterogeneous nodes in the network, as well as a list of available mesh gateways and the default mesh gateway to forward heterogeneous node data. To create this table, an OLSR routing protocol [13] is utilized to determine the route table for the Wi-Fi mesh network and employ the hop count as a metric. Then, an extension to the OLSR is added to support the use of the mesh gateway in the WMN. The extended OLSR employs two metrics to select the mesh gateway: the number of hops to the mesh gateway and the number of nodes connected to it. To achieve this, a control message is transmitted to the neighboring nodes from each mesh gateway to advertise its load in terms of the number of nodes associated with it. Each node selects the mesh gateway with the shortest path, and if more than one mesh gateway has the same number of hops, then the node selects the mesh gateway with lower load. The use of the shortest path to select the route to the mesh gateway using OLSR will avoid the occurrence of the route oscillation problem as the node utilizes only the shortest path to the mesh gateway with no switching to nonoptimal routes. Another route table is used in the mesh gateway that lists the addresses of the heterogeneous nodes associated with it. Fig. 2 shows the flowchart of creating the routing tables for each type of node. The flowchart starts by checking the node type, and then, a set of control messages is exchanged to maintain the routing table on each node. In the case of a client node with both LTE and Wi-Fi devices or HetNode, OLSR [13] is employed to create a routing table for the WMN; then, it selects the default mesh gateway based on two parameters: the distance to the mesh gateway in terms of hop count and the number of heterogeneous nodes associated with the mesh gateway. Wi-Fi devices in client nodes or HetNodes send control messages to the Internet gateway that piggyback the node Internet protocol (IP) address of the Wi-Fi network and the LTE network using the LTE transmission technology to transmit the control message to the Internet gateway through the LTE eNB base station. The Internet gateway employs this information to create a table of the Wi-Fi IP addresses and the corresponding LTE IP addresses. This table enables the Internet gateway to forward Wi-Fi packets using the LTE network, and vice versa.
In the case of mesh gateway nodes, the routing table maintains a list of the heterogeneous nodes for which it is responsible to connect them to the Internet. Each mesh gateway receives request messages from HetNodes and updates the table of HetNodes associated with it. The mesh gateways send update messages to the Internet gateway about their new list of HetNodes. Finally, nodes of type Internet gateway employ this information to maintain a table to store the available mesh gateways and the heterogeneous nodes associated with each mesh gateway. The LTE base station forwards all the Internet packets to the Internet gateway. In client nodes that are equipped with either an LTE or a Wi-Fi device, no additional routing tables are required. The LTE device directly communicates with the eNB base station, whereas the Wi-Fi device utilizes OLSR to select the mesh gateway based on hop count and load on the mesh gateway.

B. CHR Algorithm
The second part of the proposed routing protocol is the new algorithm developed, referred to here as CHR, which selects the most suitable transmission technology based on parameters from both of the utilized transmission technologies. CHR employs the generated routing tables to select the best route to send the traffic demands. The CHR is responsible for selecting the best radio access network, whereas the routing tables maintained by each node find the route to the Internet. In case a HetNode selects a Wi-Fi device, it uses the routing table to send the packets to the next hop on the path of the selected mesh gateway. CHR adopts a multirate medium access control (MAC) protocol for 802.11 called RARE [22]. RARE was developed for a WMN-only environment to consider the collision and interference in the neighboring nodes. It employs the transmission rate as a metric to measure the quality of the Wi-Fi channel. RARE reduces the transmission rate when interference is identified on the link and increases it when the interference is low. Thus, the algorithm infers that the wireless channel quality is good when the transmission rate is high. This work employs IEEE 802.11a, which supports eight different transmission rates: 6,9,12,18,24,36,48, and 54 Mb/s. A core element of CHR is the new algorithm developed to estimate which transmission technology is the best to send traffic. It is based on reinforcement learning [48] and Q-learning [48].
Reinforcement learning is a machine learning technique, which learns, through trial-and-error interactions, how to evaluate an action and find the optimal state through a mathematical formulation. The Q-learning algorithm is one of the most well-known approaches of reinforcement learning applied to wireless networks. Q-learning does not need a model of its environment; instead, it predicts the future rewards of taking an action. In Q-learning, each time (t i ) an action is executed, are w a rdR(t i ) is calculated based on feedback from the environment. Equation (1) [49] recomputes the Q-value, which is subsequently used to estimate the best action, i.e., where α is the learning rate (0 ≤ α ≤ 1), t i is the current time, t i−1 is the previous time for i>1, and γ is the discount value. If α = 0, then there is no learning in the algorithm; if γ = 0, reinforcement learning is opportunistic, which maximizes only the immediate short-term reward.
CHR, which is the algorithm proposed in this paper, is based on Q-learning to calculate whether the selected transmission technology is improving the network performance by learning from previous actions. It selects an appropriate transmission technology based on parameters from both Wi-Fi and LTE networks. The algorithm has two parts. The first part is the uplink routing algorithm, which is responsible for sending data packets from the heterogeneous nodes to the Internet. The second part is the downlink, which is in charge of transmitting data packets from the Internet to the heterogeneous nodes.
Reinforcement learning is employed in both uplink and downlink transmissions to estimate the probability of transmitting data packets through each transmission technology. For uplink transmission, each heterogeneous node utilizes CHR to select either the LTE or Wi-Fi network. In the downlink transmission, the CHR algorithm is utilized by the Internet gateway node only.
The LTE network employs both the load and the probability of successful transmissions of packets through the network as metrics to measure link quality. The load of the LTE network is estimated by measuring the buffer length for each node. This value is obtained from the radio link control (RLC) protocol layer in the eNB and the heterogeneous node. Two types of transmission buffers are maintained by the LTE network: one for downlink transmission and one for uplink transmission. Thus, the length of the buffer on each node represents its load level. Equation (2), shown below, is utilized to estimate the load on each HetNode, i.e., where LL d (t i ) is the estimated LTE load on heterogeneous node d at time t i , BufL d (t i ) is the number of packets in the LTE transmission buffer for node d at time slot t i , and BufL max is the maximum number of packets that the transmission buffer can accept. The higher LL d (t i ) is (0 <= LL d (t i ) <= 1), the more congested the node is. In WMNs, CHR employs both the transmission rate that each node utilizes to transmit its packets during time slot t i and the probability to access the channel as metrics to calculate the wireless channel quality. Equation (3), shown below, is employed to measure the Wi-Fi channel quality, i.e., where CQW d (t i ) is the Wi-Fi channel quality for node d at time t i ,a n dRW d (t i ) is the transmission rate for the Wi-Fi device at node d at time t i . According to RARE [22], which is the rate adaptation algorithm adopted by CHR, the node increases the transmission rate if the estimated interference in the neighboring nodes is low. Thus, a higher transmission rate means lower interference on the node and higher probability of sending the packets successfully. RW max is the maximum transmission rate that the WiFi transmission technology can support.
To route the packets from the heterogeneous nodes to the Internet and vice versa, the CHR algorithm is utilized for both uplink and downlink transmissions. A new algorithm based on reinforcement learning is utilized to estimate the probability of transmitting data packets through each transmission technology. Fig. 3 shows the flowchart of the CHR algorithm. The set of notations used is listed in Table I. The flowchart shows the steps of employing the CHR algorithm to utilize information maintained by each routing table generated using the proposed routing protocol. The flowchart is divided into two parts. The first part is exploration, in which the algorithm initializes the parameters used in the algorithm. Then, the learning stage starts by evaluating each action performed by the network nodes.
In particular, the algorithm starts by setting the network parameters to their defaults values, as shown in the flowchart.
The exploration stage involves sending a specific number of packets using the Wi-Fi network and the LTE network. A flag variable (FlagW) is used to indicate whether the Wi-Fi or the LTE device is being used during the exploration stage. Then, a counter variable (Expcount) is employed to control the number of explorations required to be done in this stage. The length of the exploration does not have a great impact on the system throughput, as the algorithm will converge during the reinforcement learning cycles. After finishing the exploration stage, the algorithm starts the learning stage in which each node calculates the probability of transmitting data successfully for each transmission technology by learning from previous actions using Q-learning. Equation (4), shown below, shows how Q-learning equation (1) is adapted for WMNs and incorporated in the CHR algorithm to calculate the probability of transmitting data successfully, i.e., where QW d (t i ) represents the probability of accessing the Wi-Fi channel for node d at time t i , α is the learning rate (α =0 means there is no learning in the algorithm), and SRW d (t i−1 − t i ) is the success rate of node d since the last update of the transmission rate, which is calculated using (5), shown below. CQW d (t i ) is the Wi-Fi channel quality for node d at time t i and is calculated using (3). Thus where STW d (t i−1 − t i ) is the number of successful transmissions from t i−1 until t i , which is a value that is obtained from the MAC layer of the IEEE 802.11 device on heterogeneous node d by counting the number of received acknowledgments for each transmission, and TTW d (t i−1 − t i ) is the total number of transmissions for node d using Wi-Fi from t i−1 to t i . Q-learning (1) is adopted by the CHR algorithm (6), shown below, to estimate the probability of transmitting data successfully using the LTE network, i.e., where QL d (t i ) represents the probability of accessing the LTE channel for node d at time t i ; α is the learning rate, SRL d (t i−1 − t i ) is the success rate in the LTE device of node d since the last update of the probability to access the LTE network, which is calculated using (7), shown below; and LL d (t i ) is the estimated load in the LTE device on node d at time t i and is calculated using (2). Thus where STL d (t i−1 − t i ) is the number of successful transmissions for node d during a period (t i−1 ,t i ) using the LTE network, and this information is obtained from the RLC layer using acknowledgement mode, and TTL d (t i−1 − t i ) is the number of transmissions using LTE during a period (t i−1 ,t i ).
After finishing the exploration stage, each node waits for new packets ready for transmission and then updates the probability to select the transmission technology (QL d (t i ) or QW d (t i )). Thereafter, the algorithm selects the transmission technology with the higher probability to send the packets successfully (i.e., higher Q-value). Then, CHR updates all the parameters and waits for the next packets. Here, the heterogeneous WMN is evaluated using the NS-3 simulator [50], which is a widely used tool for evaluating and validating wireless networks. In particular, this paper uses the LENA NS-3 LTE Module model [51]. The proposed network is compared in terms of throughput with LTE-only networks, Wi-Fi-only networks, and a random network (R) that randomly allocates an LTE or a Wi-Fi network for each node. Table II shows the network parameters used in the simulation. Two types of scenarios are employed to evaluate and validate the proposed network. The first scenario consists of grid topologies in which HetNodes are distributed in a grid with 100 m between each node. The second scenario consists of random topologies in which all nodes are randomly distributed in a 1000 m × 1000 m area. In both scenarios, there are five mesh gateways distributed in the network, and the LTE eNB is allocated in the center. To analyze the performance of the proposed network, different loads are applied to the network using 19 and 30 nodes transmitting simultaneously for both uplink and downlink transmissions.

B. Evaluating and Validating Results
The performance of HetMeshNet is compared with LTEonly and random networks, using different numbers of radio resource blocks (RBs), and Wi-Fi-only networks.
Two types of scenarios are employed to evaluate the proposed system: one to test the uplink and one to test the downlink. In the uplink scenarios, the nodes (except the mesh gateway nodes) generate user datagram protocol (UDP) traffic with the same rate, and the sole destination is the Internet. This simulates the uplink traffic from customer terminals to the Internet. Grid and random topologies are employed in the simulation, and two different loads are applied to the network using 19 and 30 nodes transmitting simultaneously to the Internet. A second scenario is utilized to show how the algorithm adapts to the change of the load amount during the simulation.
The simulation results for the uplink scenarios indicate significant improvement in system throughput for the proposed heterogeneous system compared with the benchmark networks.           (learning rate in reinforcement learning). If α is zero, it means that the system utilizes only the current state of the network with no learning in the system. The simulation results indicate that the network with no learning shows the worst performance in terms of throughput compared with other values of α (learning is presented). Figs. 14 and 15 show the throughput results of CHR using different values of α to demonstrate how the network works without learning, from the results α with a value higher than 0 (learning is presented in the algorithm), indicates better throughput results compared with the network with no learning (α = 0). These results indicate that considering previous network parameters in selecting the radio access technology improves the network performance. Fig. 16 shows the behavior of the network throughput at different times with different numbers of transmission nodes. In this scenario, it shows how the proposed algorithm reacts to the change of load on the network. The results indicate that the proposed algorithm outperforms the benchmark networks; for example, when the number of transmitting nodes is 15, the average throughput of the CHR is about 1.7 Mb/s with a bandwidth of 25 RB, whereas the LTE-only network with a bandwidth of 50 RB is 1 Mb/s (an increase of 70%) and that with a bandwidth of 25 RB is 0.5 Mb/s (an increase of 240%). Fig. 17 shows the behavior of the network with a constant number of client nodes that are allocated to different HetNodes with a mobility of client nodes to demonstrate how the learning algorithm react to a change in the bandwidth request. This scenario employs a random-walk mobility model to simulate the movements of client nodes in a 1000 m × 1000 m area. The results indicate that the learning algorithm adapts very well with the change in load demands in the network compared with the benchmark networks in terms of network throughput. For example, the average network throughput of CHR with a bandwidth of 25 RB is around 2 Mb/s, whereas LTE and random networks with as twice bandwidth as CHR archive around 1 and 1.5 Mb/s, respectively.
An ANOVA statistical test was performed on the results from each scenario to verify that there is a systematic enhancement in the network that causes the throughput improvement. Equation (8), shown below, is employed to confirm that the algorithms are statistically different. Thus where F is the ANOVA test statistics, and F Crit is the critical value extracted from the F-distribution table. Another parameter from the ANOVA test is p, which is the probability of having differences that happen purely by chance, and the preferred value is smaller than 0.05. Thereafter, to verify that the HetMeshNet has produced higher throughput than the benchmark algorithms, the results from each scenario are submitted to Fisher's least significant difference (LSD) test. In each scenario, there are 19 or 30 throughput results for each type of network. The average throughput value of these results is calculated for each network (LTE25 avr ,L T E 5 0 avr ,W i -F i avr ,C H R 2 5 avr , and CHR50 avr ). Next, if |CHR25 avr − LTE25 avr | > LSD, then the two averages are statistically different. Table III shows the ANOVA and LSD results for each scenario. The results show that the average throughput of the HetMeshNet outperforms the LTE network, even when the LTE network utilizes twice as much bandwidth as CHR, whereas the Wi-Fi networks may suffer from high loss due to interference and collision. In Fig. 4, the CHR algorithm with an LTE bandwidth of 25 RB achieves average uplink throughput between 1.6 and 2.3 Mb/s for 50% of the results, whereas the LTE network with 50 RB achieves between 0.7 and 1.2 Mb/s, which shows how the CHR outperforms the LTE-only network with about 183% by employing less bandwidth (half of the bandwidth), and in Fig. 8, the CHR25 increases the network throughput by about 200%.
In the downlink transmission, the throughput improvement in some scenarios is lower than that in the uplink due to the fact that in downlink, the LTE network employs a multiple-inputmultiple-output antenna that increases the total throughput of a connection in the LTE networks. For instance, Fig. 9 shows the average throughput of the CHR with 50 RB with about 1.7 Mb/s, whereas the LTE network with 50 RB achieves around 1.3 Mb/s (the improvement is about 26%). This method improves the network performance and reduces the cost of buying more licensed frequencies (LTE frequency) by utilizing unlicensed Wi-Fi frequencies instead. The results obtained from the HetMeshNet mitigate the poor performance of the Wi-Fi network through the use of the LTE network, as Wi-Fi-only networks suffer from interference. Finally, Fig. 17 shows the number of transmission packets on each transmission device and the number of packets that initially started with Wi-Fi and then switched back to the LTE network after one or more hops, for example, in node 4, about 45% of the packets are switched from a Wi-Fi network to an LTE network. This figure shows how the networks dynamically switch between the transmission technologies. The HetMeshNet improves the overall network throughput compared with the LTE network that utilizes twice as much bandwidth. Furthermore, Fig. 17 shows that many of the nodes utilize Wi-Fi bandwidth, which is cheaper than LTE because Wi-Fi frequencies are unlicensed.
The simulation experiments show that the proposed model enhances node throughput by up to 200% on the uplink and downlink compared with the LTE and Wi-Fi networks and overcomes the problem of throughput degradation in WMNs under high traffic density.

VI. CONCLUSION
This paper has introduced a new heterogeneous network architecture in which LTE and Wi-Fi wireless devices are utilized to benefit from the bandwidth of each transmission technology. In addition, a new routing protocol for heterogeneous WMNs has been developed, which dynamically selects the transmission technology to increase the overall network capacity and enhance the average throughput. Moreover, a new routing algorithm has been proposed for the needs of the routing protocol, which estimates the cost of transmitting the traffic through each network. The proposed algorithm considers the traffic load on the LTE network as a metric to estimate the cost of transmission over LTE and uses transmission rate as a metric for the Wi-Fi mesh network. The simulation results show that the proposed network achieves up to 200% more throughput compared with Wi-Fi-only networks and LTE-only networks. The heterogeneous network architecture manages the different wireless devices as a part of a single virtual network. The LTE network is utilized to avoid congested Wi-Fi nodes and a highinterference path in the WMN, whereas the WMN offloads the load of the LTE network, reduces the cost of using more licensed frequency bands, and forwards the data to another node when the LTE throughput is degrading. This work provides the foundation for future research on developing heterogeneous Wi-Fi/LTE mesh networks and using other wireless technologies as part of heterogeneous networks. The proposed routing protocol has the potential to be extended to support other wireless technologies by utilizing their parameters in the learning algorithm. The proposed architecture provides an easy way to expand the mobile network coverage and capacity and could contribute to the fifth-generation infrastructure. Moreover, the heterogeneous networks could be used to connect the Internetof-things networks and employed to provide the infrastructure for smart homes and smart cities.