VLAN keyword in platform config file will now make all traffic be sent over vlan

Signed-off-by: Ido Barnea <ibarnea@cisco.com>

21 files changed, 259 insertions, 90 deletions

diff --git a/.gitignore b/.gitignore
index 054905ce..bed1d2d4 100644
--- a/.gitignore
+++ b/.gitignore
@@ -82,9 +82,9 @@ scripts/exp/stl_vm_enable0_cache_10-0.erf
 scripts/exp/stl_vm_enable1_cache_500-0.erf
 scripts/exp/imix-0.erf
 scripts/exp/imix_v6-0.erf
-scripts/exp/ipv4_vlan-0.erf
+scripts/exp/ipv4_load_balance-0.erf
 scripts/exp/ipv6-0.erf
-scripts/exp/ipv6_vlan-0.erf
+scripts/exp/ipv6_load_balance-0.erf
 scripts/exp/limit_multi_pkt-0.erf
 scripts/exp/limit_single_pkt-0.erf
 scripts/exp/imix_64_fast-0.erf
diff --git a/doc/trex_book.asciidoc b/doc/trex_book.asciidoc
index 366102b8..c59aa1f1 100755
--- a/doc/trex_book.asciidoc
+++ b/doc/trex_book.asciidoc
@@ -664,15 +664,30 @@ include::trex_book_basic.asciidoc[]
 
 == Advanced features
 
+=== VLAN (dot1q) support
+
+If you want VLAN tag to be added to all traffic generated by TRex, you can acheive that by adding ``vlan'' keyword in each
+port section in the platform config file, like described xref:trex_config[here]. +
+You can specify different VLAN tag for each port, or even use VLAN only on some of the ports. +
+One useful application of this can be in a lab setup where you have one TRex and many DUTs, and you want to test different
+DUT on each run, without changing cable connections. You can put each DUT on a VLAN of its own, and use different TRex
+platform config files with different VLANs on each run.
+
+
 === Utilizing maximum port bandwidth in case of asymmetric traffic profile 
 
-anchor:trex_vlan[]
+anchor:trex_load_bal[]
+
+[NOTE]
+If you want simple VLAN support, this is probably *not* the feature you want. This is used for load balancing.
+If you want VLAN support, please look at ``vlan'' field xref:trex_config[here].
+
 
 The VLAN Trunk TRex feature attempts to solve the router port bandwidth limitation when the traffic profile is asymmetric. Example: Asymmetric SFR profile.
 This feature converts asymmetric traffic to symmetric, from the port perspective, using router sub-interfaces.
 This requires TRex to send the traffic on two VLANs, as described below.
 
-.YAML format 
+.YAML format - This goes into traffic yaml file
 [source,python]
 ----
   vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
@@ -685,27 +700,29 @@ This requires TRex to send the traffic on two VLANs, as described below.
 - duration : 0.1
   vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }   <1>
 ----
-<1> Enable VLAN feature, vlan0==100 , vlan1==200
-        
+<1> Enable load balance feature, vlan0==100 , vlan1==200
+For a full file example please look in TRex source at scripts/cap2/ipv4_load_balance.yaml
+
 *Problem definition:*::
 
 Scenario: TRex with two ports and an SFR traffic profile.
 
-.Without VLAN/sub interfaces 
+.Without VLAN/sub interfaces, all client emulated traffic is sent on port 0, and all server emulated traffic (HTTP response for example) on port 1.
 [source,python]
 ----
-0 ( client) -> [  ] - 1 ( server)
+TRex port 0 ( client) <-> [  DUT ] <-> TRex port 1 ( server)
 ----
-Without VLAN support the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% is from port 1 (server). Port 1 become the bottlneck (10Gb/s limit) before port 0.
+Without VLAN support the traffic is asymmetric. 10% of the traffic is sent from port 0 (client side), 90% is from port 1 (server). Port 1 is the bottlneck (10Gb/s limit).
 
 .With VLAN/sub interfaces 
 [source,python]
 ----
-port 0 ( client VLAN0) <->  |  | <-> port 1 ( server-VLAN0)
-port 0 ( server VLAN1) <->  |  | <-> port 1 ( client-VLAN1)
+TRex port 0 ( client VLAN0) <->  | DUT  | <-> TRex port 1 ( server-VLAN0)
+TRex port 0 ( server VLAN1) <->  | DUT  | <-> TRex port 1 ( client-VLAN1)
 ----
 
-In this case both ports have the same amount of traffic.
+In this case, traffic on vlan0 is sent as before, while for traffic on vlan1, order is reversed (client traffic sent on port1 and server traffic on port0).
+TRex divids the flows evenly between the vlans. This results an equal amount of traffic on each port.
 
 *Router configuation:*::
 [source,python]
@@ -766,8 +783,7 @@ In this case both ports have the same amount of traffic.
          set ip next-hop 11.88.11.12
         !
 ----
-<1> Disable the IP on the main port it is important.
-// above, clarify what's important
+<1> Main interface must not have IP address.
 <2> Enable VLAN1
 <3> PBR configuration
 <4> Enable VLAN2  
@@ -1307,7 +1323,7 @@ Cpu Utilization : 0.1 %
 <4> true (default) indicates that the IPG is taken from the cap file (also taking into account cap_ipg_min and cap_override_ipg if they exist). false indicates that IPG is taken from per template section.
 <5> The following two options can set the min ipg in microseconds: (if (pkt_ipg<cap_ipg_min) { pkt_ipg=cap_override_ipg} )
 <6> Value to override (microseconds), as described in note above.
-<7> Enable vlan feature. See xref:trex_vlan[trex_vlan section] for info.
+<7> Enable load balance feature. See xref:trex_load_bal[trex load balance section] for info.
 <8> Enable MAC address replacement by client IP.
 
 
@@ -1397,7 +1413,7 @@ the result as dest MAC. If no dest_mac given, and no ARP response received, TRex
 <11> Source MAC to use when sending packets from this interface. If not given (since version 2.10), MAC address of the port will be used.
 <12> If given (since version 2.10), TRex will issue gratitues ARP for the ip + src MAC pair on appropriate port. In stateful mode,
 gratitues ARP for each ip will be sent every 120 seconds (Can be changed using --arp-refresh-period argument).
-<13> If given, gratitues ARP and ARP request will be sent using the given VLAN tag.
+<13> If given (since version 2.18), all traffic on the port will be sent with this VLAN tag.
 <14> Old MAC address format. New format is supported since version v2.09.
 
 [NOTE]
diff --git a/scripts/automation/regression/setups/trex25/benchmark.yaml b/scripts/automation/regression/setups/trex25/benchmark.yaml
index 2c677b81..7f357ce1 100644
--- a/scripts/automation/regression/setups/trex25/benchmark.yaml
+++ b/scripts/automation/regression/setups/trex25/benchmark.yaml
@@ -107,6 +107,10 @@ test_client_cfg_vlan:
     cores               : 1
     multiplier          : 10
 
+test_platform_cfg_vlan:
+    cores               : 1
+    multiplier          : 10
+
 test_rx_check_http: &rx_http
     multiplier          : 8800
     cores               : 1
diff --git a/scripts/automation/regression/stateful_tests/trex_vlan_test.py b/scripts/automation/regression/stateful_tests/trex_vlan_test.py
new file mode 100644
index 00000000..13d89565
--- /dev/null
+++ b/scripts/automation/regression/stateful_tests/trex_vlan_test.py
@@ -0,0 +1,83 @@
+#!/router/bin/python
+from .trex_general_test import CTRexGeneral_Test, CTRexScenario
+from .tests_exceptions import *
+import time
+from CPlatform import CStaticRouteConfig
+
+class CTRexVlan_Test(CTRexGeneral_Test):#(unittest.TestCase):
+    """This class defines test for vlan platform configutation file"""
+    def __init__(self, *args, **kwargs):
+        super(CTRexVlan_Test, self).__init__(*args, **kwargs)
+        self.unsupported_modes = ['loopback'] # We test on routers
+
+    def setUp(self):
+        super(CTRexVlan_Test, self).setUp() # launch super test class setUp process
+
+    def test_platform_cfg_vlan(self):
+        if self.is_loopback:
+            self.skip('Not relevant on loopback')
+
+        if not CTRexScenario.router_cfg['no_dut_config']:
+            self.router.configure_basic_interfaces(vlan = True)
+            self.router.config_pbr(mode = "config", vlan = True)
+
+        mult = self.get_benchmark_param('multiplier')
+        core = self.get_benchmark_param('cores')
+
+        self.create_vlan_conf_file()
+
+        ret = self.trex.start_trex (
+            c = core,
+            m = mult,
+            d = 60,
+            f = 'cap2/dns.yaml',
+            cfg = '/tmp/trex_files/trex_cfg_vlan.yaml',
+            l = 100,
+            limit_ports = 4)
+
+        trex_res = self.trex.sample_to_run_finish()
+        print("\nLATEST RESULT OBJECT:")
+        print(trex_res)
+        self.check_general_scenario_results(trex_res, check_latency = True)
+
+    def tearDown(self):
+        pass
+
+    # Add vlan to platform config file
+    # write the result to new config file in /tmp, to be used by the test
+    def create_vlan_conf_file(self):
+
+        remote_cfg_file = self.trex.get_trex_config()
+        out = open("/tmp/trex_cfg_vlan.yaml", "w")
+        insert_vlan = False
+        vlan_val_1 = "100"
+        vlan_val_2 = "200"
+        vlan_val = vlan_val_1
+
+        for line in remote_cfg_file.split("\n"):
+            out.write (line + "\n")
+            # when we see "port_info" start inserting, until port_info section end
+            if "port_info" in line.lstrip():
+                num_space_port_info = len(line) - len(line.lstrip())
+                insert_vlan = True
+                continue
+
+            if insert_vlan:
+                num_space = len(line) - len(line.lstrip())
+                if num_space == num_space_port_info:
+                    insert_vlan = False
+                    continue
+                if len(line.lstrip()) > 0 and line.lstrip()[0] == '-':
+                    # insert vlan for each port (every time a section start with '-'
+                    num_space = len(line) - len(line.lstrip(' -'))
+                    # need to insert in the same indentation as other lines in the section
+                    out.write(num_space* " " + "vlan: " + vlan_val + "\n")
+                    if vlan_val == vlan_val_1:
+                        vlan_val = vlan_val_2                            
+                    else:
+                        vlan_val = vlan_val_1
+        out.close()
+        self.trex.push_files("/tmp/trex_cfg_vlan.yaml");
+                        
+if __name__ == "__main__":
+    pass
diff --git a/scripts/automation/trex_control_plane/stf/trex_stf_lib/trex_client.py b/scripts/automation/trex_control_plane/stf/trex_stf_lib/trex_client.py
index 490e3b7a..43504c96 100755
--- a/scripts/automation/trex_control_plane/stf/trex_stf_lib/trex_client.py
+++ b/scripts/automation/trex_control_plane/stf/trex_stf_lib/trex_client.py
@@ -889,6 +889,31 @@ class CTRexClient(object):
         finally:
             self.prompt_verbose_data()
 
+    def get_trex_config(self):
+        """
+        Get Trex config file (/etc/trex_cfg.yaml).
+
+        :return:
+            String representation of TRex config file
+
+        :raises:
+            + :exc:`trex_exceptions.TRexRequestDenied`, in case file could not be read.
+            + ProtocolError, in case of error in JSON-RPC protocol.
+
+        """
+        try:
+            res = binascii.a2b_base64(self.server.get_trex_config())
+            if type(res) is bytes:
+                return res.decode()
+            return res
+        except AppError as err:
+            self._handle_AppError_exception(err.args[0])
+        except ProtocolError:
+            raise
+        finally:
+            self.prompt_verbose_data()
+
+
     def push_files (self, filepaths):
         """
         Pushes a file (or a list of files) to store locally on server. 
diff --git a/scripts/cap2/dns.yaml b/scripts/cap2/dns.yaml
index dd577894..23b02fd6 100755
--- a/scripts/cap2/dns.yaml
+++ b/scripts/cap2/dns.yaml
@@ -11,8 +11,6 @@
           tcp_aging      : 1
           udp_aging      : 1
   mac        : [0x00,0x00,0x00,0x01,0x00,0x00]
-  #vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
-  #mac_override_by_ip : true
   cap_info : 
      - name: cap2/dns.pcap
        cps : 1.0
diff --git a/scripts/cap2/dns_no_delay.yaml b/scripts/cap2/dns_no_delay.yaml
index ab387d31..6139716c 100644
--- a/scripts/cap2/dns_no_delay.yaml
+++ b/scripts/cap2/dns_no_delay.yaml
@@ -11,8 +11,6 @@
           tcp_aging      : 0
           udp_aging      : 0
   mac        : [0x00,0x00,0x00,0x01,0x00,0x00]
-  #vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
-  #mac_override_by_ip : true
   cap_info : 
      - name: cap2/dns.pcap
        cps : 1.0
diff --git a/scripts/cap2/ipv4_vlan.yaml b/scripts/cap2/ipv4_load_balance.yaml
index 63f7db7d..63f7db7d 100755
--- a/scripts/cap2/ipv4_vlan.yaml
+++ b/scripts/cap2/ipv4_load_balance.yaml
diff --git a/scripts/cap2/ipv6_vlan.yaml b/scripts/cap2/ipv6_load_balance.yaml
index bb91a4f8..bb91a4f8 100755
--- a/scripts/cap2/ipv6_vlan.yaml
+++ b/scripts/cap2/ipv6_load_balance.yaml
diff --git a/scripts/cap2/jumbo.yaml b/scripts/cap2/jumbo.yaml
index b45a6ca3..901e6189 100644
--- a/scripts/cap2/jumbo.yaml
+++ b/scripts/cap2/jumbo.yaml
@@ -11,8 +11,6 @@
           tcp_aging      : 1
           udp_aging      : 1
   mac        : [0x00,0x00,0x00,0x01,0x00,0x00]
-  #vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
-  #mac_override_by_ip : true
   cap_info : 
      - name: cap2/jumbo.pcap
        cps : 1.0
diff --git a/scripts/cap2/wrong_ip.yaml b/scripts/cap2/wrong_ip.yaml
index 7de3b82d..226b0c5c 100644
--- a/scripts/cap2/wrong_ip.yaml
+++ b/scripts/cap2/wrong_ip.yaml
@@ -11,8 +11,6 @@
           tcp_aging      : 1
           udp_aging      : 1
   mac        : [0x00,0x00,0x00,0x01,0x00,0x00]
-  #vlan       : { enable : 1  ,  vlan0 : 100 , vlan1 : 200 }
-  #mac_override_by_ip : true
   cap_info : 
      - name: cap2/wrong_ip.pcap
        cps : 1.0
diff --git a/scripts/exp/ipv4_vlan-0-ex.erf b/scripts/exp/ipv4_load_balance-0-ex.erf
index 435d8c32..435d8c32 100755
--- a/scripts/exp/ipv4_vlan-0-ex.erf
+++ b/scripts/exp/ipv4_load_balance-0-ex.erf
diff --git a/scripts/exp/ipv4_vlan-0.erf b/scripts/exp/ipv4_vlan-0.erf
deleted file mode 100644
index 435d8c32..00000000
--- a/scripts/exp/ipv4_vlan-0.erf
+++ /dev/null
diff --git a/scripts/exp/ipv6_vlan-0-ex.erf b/scripts/exp/ipv6_load_balance-0-ex.erf
index 8ca5f09b..8ca5f09b 100755
--- a/scripts/exp/ipv6_vlan-0-ex.erf
+++ b/scripts/exp/ipv6_load_balance-0-ex.erf
diff --git a/scripts/exp/ipv6_vlan-0.erf b/scripts/exp/ipv6_vlan-0.erf
deleted file mode 100644
index 8ca5f09b..00000000
--- a/scripts/exp/ipv6_vlan-0.erf
+++ /dev/null
diff --git a/src/bp_gtest.cpp b/src/bp_gtest.cpp
index 57cf2ffa..e1bfeeab 100755
--- a/src/bp_gtest.cpp
+++ b/src/bp_gtest.cpp
@@ -449,8 +449,8 @@ TEST_F(basic, ipv4_vlan) {
      CParserOption * po =&CGlobalInfo::m_options;
      po->preview.setVMode(3);
      po->preview.setFileWrite(true);
-     po->cfg_file ="cap2/ipv4_vlan.yaml";
-     po->out_file ="exp/ipv4_vlan";
+     po->cfg_file ="cap2/ipv4_load_balance.yaml";
+     po->out_file ="exp/ipv4_load_balance";
      bool res=t1.init();
      EXPECT_EQ_UINT32(1, res?1:0)<< "pass";
 }
@@ -462,8 +462,8 @@ TEST_F(basic, ipv6_vlan) {
      po->preview.setVMode(3);
      po->preview.set_ipv6_mode_enable(true);
      po->preview.setFileWrite(true);
-     po->cfg_file ="cap2/ipv6_vlan.yaml";
-     po->out_file ="exp/ipv6_vlan";
+     po->cfg_file ="cap2/ipv6_load_balance.yaml";
+     po->out_file ="exp/ipv6_load_balance";
      bool res=t1.init();
      EXPECT_EQ_UINT32(1, res?1:0)<< "pass";
 }
diff --git a/src/bp_sim.cpp b/src/bp_sim.cpp
index f540f21b..d3c8abbd 100755
--- a/src/bp_sim.cpp
+++ b/src/bp_sim.cpp
@@ -758,12 +758,23 @@ void CPreviewMode::Dump(FILE *fd){
     fprintf(fd," flow-flip       : %d\n", (int)getClientServerFlowFlip() );
     fprintf(fd," no clean close  : %d\n", (int)getNoCleanFlowClose() );
     fprintf(fd," zmq_publish     : %d\n", (int)get_zmq_publish_enable() );
-    fprintf(fd," vlan_enable     : %d\n", (int)get_vlan_mode_enable() );
+    fprintf(fd," vlan mode       : %d\n",  get_vlan_mode());
     fprintf(fd," client_cfg      : %d\n", (int)get_is_client_cfg_enable() );
     fprintf(fd," mbuf_cache_disable  : %d\n", (int)isMbufCacheDisabled() );
     fprintf(fd," vm mode         : %d\n", (int)get_vm_one_queue_enable()?1:0 );
 }
 
+void CPreviewMode::set_vlan_mode_verify(uint8_t mode) {
+        // validate that there is no vlan both in platform config file and traffic profile
+    if ((CGlobalInfo::m_options.preview.get_vlan_mode() != CPreviewMode::VLAN_MODE_NONE) &&
+        ( CGlobalInfo::m_options.preview.get_vlan_mode() != mode ) ) {
+        fprintf(stderr, "Error: You are not allowed to specify vlan both in platform config file (--cfg) and traffic config file (-f)\n");
+        fprintf(stderr, "       Please remove vlan definition from one of the files, and try again.\n");
+        exit(1);
+    }
+    set_vlan_mode(mode);
+}
+
 void CFlowGenStats::clear(){
    m_nat_lookup_no_flow_id=0;
    m_total_bytes=0;
@@ -5060,9 +5071,14 @@ int CFlowGenList::load_from_yaml(std::string file_name,
     }
 
     /* move it to global info, better CPU D-cache  usage */
-    CGlobalInfo::m_options.preview.set_vlan_mode_enable(m_yaml_info.m_vlan_info.m_enable);
-    CGlobalInfo::m_options.m_vlan_port[0] =   m_yaml_info.m_vlan_info.m_vlan_per_port[0];
-    CGlobalInfo::m_options.m_vlan_port[1] =   m_yaml_info.m_vlan_info.m_vlan_per_port[1];
+    if (m_yaml_info.m_vlan_info.m_enable) {
+        CGlobalInfo::m_options.preview.set_vlan_mode_verify(CPreviewMode::VLAN_MODE_LOAD_BALANCE);
+        CGlobalInfo::m_options.m_vlan_port[0] = m_yaml_info.m_vlan_info.m_vlan_per_port[0];
+        CGlobalInfo::m_options.m_vlan_port[1] = m_yaml_info.m_vlan_info.m_vlan_per_port[1];
+    } else {
+        CGlobalInfo::m_options.m_vlan_port[0] = 0;
+        CGlobalInfo::m_options.m_vlan_port[1] = 0;
+    }
     CGlobalInfo::m_options.preview.set_mac_ip_overide_enable(m_yaml_info.m_mac_replace_by_ip);
 
     if ( m_yaml_info.m_mac_base.size() != 6 ){
@@ -5386,8 +5402,8 @@ void CParserOption::dump(FILE *fd){
     fprintf(fd," tw_levels       : %lu usec \n",(ulong)get_tw_levels());   
 
 
-    if (preview.get_vlan_mode_enable() ) {
-       fprintf(fd," vlans     : [%d,%d] \n",m_vlan_port[0],m_vlan_port[1]);
+    if (preview.get_vlan_mode() == CPreviewMode::VLAN_MODE_LOAD_BALANCE) {
+       fprintf(fd," vlans (for load balance) : [%d,%d] \n",m_vlan_port[0],m_vlan_port[1]);
     }
 
     int i;
@@ -5397,15 +5413,23 @@ void CParserOption::dump(FILE *fd){
         dump_mac_addr(fd,lp->u.m_mac.dest);
         fprintf(fd,"  src:");
         dump_mac_addr(fd,lp->u.m_mac.src);
+        if (preview.get_vlan_mode() == CPreviewMode::VLAN_MODE_NORMAL) {
+            fprintf(fd, " vlan:%d", m_ip_cfg[i].get_vlan());
+        }
         fprintf(fd,"\n");
     }
+
 }
 
 void CParserOption::verify() {
     /* check for mutual exclusion options */
-    if (preview.get_is_client_cfg_enable()) {
-        if (preview.get_vlan_mode_enable() || preview.get_mac_ip_overide_enable()) {
-            throw std::runtime_error("VLAN / MAC override cannot be combined with client configuration");
+    if ( preview.get_is_client_cfg_enable() ) {
+        if ( preview.get_vlan_mode() == CPreviewMode::VLAN_MODE_LOAD_BALANCE ) {
+            throw std::runtime_error("--client_cfg_file option can not be combined with specifing VLAN in traffic profile");
+        }
+
+        if (preview.get_mac_ip_overide_enable()) {
+            throw std::runtime_error("MAC override can not be combined with --client_cfg_file option");
         }
     }
 }
@@ -5754,9 +5778,11 @@ int CErfIF::send_node(CGenNode *node){
     if (CGlobalInfo::m_options.preview.get_is_client_cfg_enable()) {
         apply_client_config(node->m_client_cfg, dir);
 
-    } else if (CGlobalInfo::m_options.preview.get_vlan_mode_enable()) {
+        // for simulation, VLAN_MODE_NORMAL is not relevant, since it uses vlan_id set in platform config file
+    } else if (CGlobalInfo::m_options.preview.get_vlan_mode() == CPreviewMode::VLAN_MODE_LOAD_BALANCE) {
         uint8_t vlan_port = (node->m_src_ip & 1);
         uint16_t vlan_id = CGlobalInfo::m_options.m_vlan_port[vlan_port];
+
         add_vlan(vlan_id);
     }
 
diff --git a/src/bp_sim.h b/src/bp_sim.h
index 22968b2c..bc2eaf03 100755
--- a/src/bp_sim.h
+++ b/src/bp_sim.h
@@ -386,14 +386,20 @@ public:
 
 class CPreviewMode {
 public:
+    enum {
+        VLAN_MODE_NONE = 0,
+        VLAN_MODE_NORMAL = 1,
+        VLAN_MODE_LOAD_BALANCE = 2,
+    };
+
     CPreviewMode(){
         clean();
     }
     void clean(){
         m_flags = 0;
         m_flags1=0;
-        set_vlan_mode_enable(false);
         setCores(1);
+        set_vlan_mode(VLAN_MODE_NONE);
         set_zmq_publish_enable(true);
     }
 
@@ -511,7 +517,13 @@ public:
         return (btGetMaskBit32(m_flags,25,25) ? true:false);
     }
 
-    // bit 26 is free. Was deprecated option.
+    void set_pcap_mode_enable(bool enable){
+        btSetMaskBit32(m_flags,26,26,enable?1:0);
+    }
+
+    bool get_pcap_mode_enable(){
+        return (btGetMaskBit32(m_flags,26,26) ? true:false);
+    }
 
     void set_zmq_publish_enable(bool enable){
         btSetMaskBit32(m_flags,27,27,enable?1:0);
@@ -521,23 +533,15 @@ public:
         return (btGetMaskBit32(m_flags,27,27) ? true:false);
     }
 
-    void set_pcap_mode_enable(bool enable){
-        btSetMaskBit32(m_flags,28,28,enable?1:0);
-    }
-
-    bool get_pcap_mode_enable(){
-        return (btGetMaskBit32(m_flags,28,28) ? true:false);
-    }
-
-    /* VLAN enable/disable */
-    bool get_vlan_mode_enable(){
-        return (btGetMaskBit32(m_flags,29,29) ? true:false);
+    uint8_t get_vlan_mode() {
+        return (btGetMaskBit32(m_flags, 29, 28));
     }
 
-    void set_vlan_mode_enable(bool enable){
-        btSetMaskBit32(m_flags,29,29,enable?1:0);
+    void set_vlan_mode(uint8_t mode) {
+        btSetMaskBit32(m_flags, 29, 28, mode);
     }
 
+    void set_vlan_mode_verify(uint8_t mode);
     bool get_mac_ip_overide_enable(){
         return (btGetMaskBit32(m_flags,30,30) ? true:false);
     }
diff --git a/src/main_dpdk.cpp b/src/main_dpdk.cpp
index 5c9d2498..4e848f9e 100644
--- a/src/main_dpdk.cpp
+++ b/src/main_dpdk.cpp
@@ -1229,7 +1229,8 @@ static int parse_options(int argc, char *argv[], CParserOption* po, bool first_t
 
     if ( get_is_stateless() ) {
         if ( opt_vlan_was_set ) {
-            po->preview.set_vlan_mode_enable(true);
+            // Only purpose of this in stateless is for configuring the 82599 rules correctly
+            po->preview.set_vlan_mode(CPreviewMode::VLAN_MODE_NORMAL);
         }
         if (CGlobalInfo::m_options.client_cfg_file != "") {
             parse_err("Client config file is not supported with interactive (stateless) mode ");
@@ -1355,7 +1356,7 @@ public:
         if (get_is_stateless()) {
             m_port_conf.fdir_conf.flexbytes_offset = (14+4)/2;
             /* Increment offset 4 bytes for the case where we add VLAN */
-            if (  CGlobalInfo::m_options.preview.get_vlan_mode_enable() ) {
+            if (  CGlobalInfo::m_options.preview.get_vlan_mode() != CPreviewMode::VLAN_MODE_NONE) {
                 m_port_conf.fdir_conf.flexbytes_offset += (4/2);
             }
         } else {
@@ -1366,7 +1367,7 @@ public:
             }
 
             /* Increment offset 4 bytes for the case where we add VLAN */
-            if (  CGlobalInfo::m_options.preview.get_vlan_mode_enable() ) {
+            if (  CGlobalInfo::m_options.preview.get_vlan_mode() != CPreviewMode::VLAN_MODE_NONE ) {
                 m_port_conf.fdir_conf.flexbytes_offset += (4/2);
             }
         }
@@ -2152,8 +2153,6 @@ protected:
                  rte_mbuf_t *m,
                  CVirtualIFPerSideStats  * lp_stats);
 
-    void add_vlan(rte_mbuf_t *m, uint16_t vlan_id);
-
 protected:
     uint8_t      m_core_id;
     uint16_t     m_mbuf_cache;
@@ -2545,13 +2544,6 @@ void CCoreEthIF::apply_client_cfg(const ClientCfgBase *cfg, rte_mbuf_t *m, pkt_d
     }
 }
 
-
-void CCoreEthIF::add_vlan(rte_mbuf_t *m, uint16_t vlan_id) {
-    m->ol_flags = PKT_TX_VLAN_PKT;
-    m->l2_len   = 14;
-    m->vlan_tci = vlan_id;
-}
-
 /**
  * slow path features goes here (avoid multiple IFs)
  *
@@ -2600,17 +2592,26 @@ int CCoreEthIF::send_node(CGenNode * node) {
     single_port = node->get_is_all_flow_from_same_dir() ;
 
 
-    if ( unlikely( CGlobalInfo::m_options.preview.get_vlan_mode_enable() ) ){
-        /* which vlan to choose 0 or 1*/
-        uint8_t vlan_port = (node->m_src_ip &1);
-        uint16_t vlan_id  = CGlobalInfo::m_options.m_vlan_port[vlan_port];
+    if ( unlikely(CGlobalInfo::m_options.preview.get_vlan_mode()
+                  != CPreviewMode::VLAN_MODE_NONE) ) {
+        uint16_t vlan_id=0;
 
-        if (likely( vlan_id >0 ) ) {
-            dir = dir ^ vlan_port;
-        }else{
-            /* both from the same dir but with VLAN0 */
-            vlan_id = CGlobalInfo::m_options.m_vlan_port[0];
-            dir = dir ^ 0;
+        if (CGlobalInfo::m_options.preview.get_vlan_mode()
+            == CPreviewMode::VLAN_MODE_LOAD_BALANCE) {
+            /* which vlan to choose 0 or 1*/
+            uint8_t vlan_port = (node->m_src_ip & 1);
+            vlan_id = CGlobalInfo::m_options.m_vlan_port[vlan_port];
+            if (likely( vlan_id > 0 ) ) {
+                dir = dir ^ vlan_port;
+            } else {
+                /* both from the same dir but with VLAN0 */
+                vlan_id = CGlobalInfo::m_options.m_vlan_port[0];
+            }
+        } else if (CGlobalInfo::m_options.preview.get_vlan_mode()
+            == CPreviewMode::VLAN_MODE_NORMAL) {
+            CCorePerPort *lp_port = &m_ports[dir];
+            uint8_t port_id = lp_port->m_port->get_port_id();
+            vlan_id = CGlobalInfo::m_options.m_ip_cfg[port_id].get_vlan();
         }
 
         add_vlan(m, vlan_id);
@@ -2704,12 +2705,14 @@ public:
         rte_mbuf_t *tx_pkts[2];
         
         tx_pkts[0] = m;
-        if ( likely( CGlobalInfo::m_options.preview.get_vlan_mode_enable() ) ){
-            /* vlan mode is the default */
-            /* set the vlan */
-            m->ol_flags = PKT_TX_VLAN_PKT;
-            m->vlan_tci =CGlobalInfo::m_options.m_vlan_port[0];
-            m->l2_len   =14;
+        uint8_t vlan_mode = CGlobalInfo::m_options.preview.get_vlan_mode();
+        if ( likely( vlan_mode != CPreviewMode::VLAN_MODE_NONE) ) {
+            if ( vlan_mode == CPreviewMode::VLAN_MODE_LOAD_BALANCE ) {
+                add_vlan(m, CGlobalInfo::m_options.m_vlan_port[0]);
+            } else if (vlan_mode == CPreviewMode::VLAN_MODE_NORMAL) {
+                uint8_t port_id = m_port->get_rte_port_id();
+                add_vlan(m, CGlobalInfo::m_options.m_ip_cfg[port_id].get_vlan());
+            }
         }
         uint16_t res=m_port->tx_burst(m_tx_queue_id,tx_pkts,1);
         if ( res == 0 ) {
@@ -2799,12 +2802,15 @@ public:
 private:
       virtual int tx_common(rte_mbuf_t *m, bool fix_timestamp) {
         
-        if ( likely( CGlobalInfo::m_options.preview.get_vlan_mode_enable() ) ){
-            /* vlan mode is the default */
-            /* set the vlan */
-            m->ol_flags = PKT_TX_VLAN_PKT;
-            m->vlan_tci =CGlobalInfo::m_options.m_vlan_port[0];
-            m->l2_len   =14;
+
+        uint8_t vlan_mode = CGlobalInfo::m_options.preview.get_vlan_mode();
+        if ( likely( vlan_mode != CPreviewMode::VLAN_MODE_NONE) ) {
+            if ( vlan_mode == CPreviewMode::VLAN_MODE_LOAD_BALANCE ) {
+                add_vlan(m, CGlobalInfo::m_options.m_vlan_port[0]);
+            } else if (vlan_mode == CPreviewMode::VLAN_MODE_NORMAL) {
+                uint8_t port_id = m_port->get_rte_port_id();
+                add_vlan(m, CGlobalInfo::m_options.m_ip_cfg[port_id].get_vlan());
+            }
         }
 
         /* allocate node */
@@ -4002,7 +4008,7 @@ bool CGlobalTRex::Create(){
     }
 
     if ( pre_yaml_info.m_vlan_info.m_enable ){
-        CGlobalInfo::m_options.preview.set_vlan_mode_enable(true);
+        CGlobalInfo::m_options.preview.set_vlan_mode_verify(CPreviewMode::VLAN_MODE_LOAD_BALANCE);
     }
     /* End update pre flags */
 
@@ -5514,6 +5520,10 @@ int update_global_info_from_platform_file(){
             CGlobalInfo::m_options.m_ip_cfg[i].set_ip(cg->m_mac_info[i].get_ip());
             CGlobalInfo::m_options.m_ip_cfg[i].set_mask(cg->m_mac_info[i].get_mask());
             CGlobalInfo::m_options.m_ip_cfg[i].set_vlan(cg->m_mac_info[i].get_vlan());
+            // If one of the ports has vlan, work in vlan mode
+            if (cg->m_mac_info[i].get_vlan() != 0) {
+                CGlobalInfo::m_options.preview.set_vlan_mode_verify(CPreviewMode::VLAN_MODE_NORMAL);
+            }
         }
     }
 
@@ -6145,7 +6155,7 @@ int CTRexExtendedDriverBase1G::configure_rx_filter_rules_statefull(CPhyEthIF * _
             _if->pci_reg_write( (E1000_FHFT(rule_id)+i) , 0);
         }
 
-        if (  CGlobalInfo::m_options.preview.get_vlan_mode_enable() ){
+        if (CGlobalInfo::m_options.preview.get_vlan_mode() != CPreviewMode::VLAN_MODE_NONE) {
             len += 8;
             if (  CGlobalInfo::m_options.preview.get_ipv6_mode_enable() ){
                 // IPv6 VLAN: NextHdr/HopLimit offset = 0x18
@@ -6544,7 +6554,8 @@ int CTRexExtendedDriverBase10G::wait_for_stable_link(){
 }
 
 CFlowStatParser *CTRexExtendedDriverBase10G::get_flow_stat_parser() {
-    CFlowStatParser *parser = new C82599Parser(CGlobalInfo::m_options.preview.get_vlan_mode_enable() ? true:false);
+    CFlowStatParser *parser = new C82599Parser((CGlobalInfo::m_options.preview.get_vlan_mode()
+                                                != CPreviewMode::VLAN_MODE_NONE) ? true:false);
     assert (parser);
     return parser;
 }
diff --git a/src/pal/common/common_mbuf.h b/src/pal/common/common_mbuf.h
index c52842bd..3816d203 100644
--- a/src/pal/common/common_mbuf.h
+++ b/src/pal/common/common_mbuf.h
@@ -17,6 +17,12 @@ See the License for the specific language governing permissions and
 limitations under the License.
 */
 
+static inline void add_vlan(rte_mbuf_t *m, uint16_t vlan_id) {
+    m->ol_flags = PKT_TX_VLAN_PKT;
+    m->l2_len   = 14;
+    m->vlan_tci = vlan_id;
+}
+
 static inline rte_mbuf_t * utl_rte_pktmbuf_add_after2(rte_mbuf_t *m1,rte_mbuf_t *m2){
     utl_rte_pktmbuf_check(m1);
     utl_rte_pktmbuf_check(m2);
diff --git a/src/pal/linux/mbuf.h b/src/pal/linux/mbuf.h
index 2996b514..17b7c7ad 100755
--- a/src/pal/linux/mbuf.h
+++ b/src/pal/linux/mbuf.h
@@ -32,6 +32,7 @@ typedef struct rte_mbuf  rte_mbuf_t;
 #define MAGIC2 0x11223344
 
 #define IND_ATTACHED_MBUF    (1ULL << 62) /**< Indirect attached mbuf */
+#define PKT_TX_VLAN_PKT      (1ULL << 57) /**< TX packet is a 802.1q VLAN packet. */
 #define RTE_MBUF_INDIRECT(mb)   ((mb)->ol_flags & IND_ATTACHED_MBUF)
 #define RTE_MBUF_TO_BADDR(mb)       (((struct rte_mbuf *)(mb)) + 1)
 #define RTE_MBUF_FROM_BADDR(ba)     (((struct rte_mbuf *)(ba)) - 1)
@@ -65,6 +66,7 @@ struct rte_mbuf {
     uint64_t ol_flags;        /**< Offload features. */
     uint16_t l2_len;
     uint16_t l3_len;
+    uint16_t vlan_tci;
 } ;
 
 typedef struct rte_mempool rte_mempool_t;