| Age | Commit message (Collapse) | Author | Files | Lines |
|---|
|
A bug was reported where a jumbo packet would stay in vhost
queue forever or until a large enough number of other packets
arrived in the queue too.
This is due to a bug in vhost input node buffer allocation.
The fix is to make sure that vhost always allocates at least
enough buffers for one single big packet. '40' is used to
account for 65kB frames.
Change-Id: I1d293028854165083e30cd798fab9d4140230b78
Signed-off-by: Pierre Pfister <ppfister@cisco.com>
|
|
With heavy traffic, tx code path may crash due to memory corruption
Thread 5 "vpp_wk_2" received signal SIGSEGV, Segmentation fault.
[Switching to Thread 0x7fff3995c700 (LWP 2505)]
0x00007ffff73675e8 in vhost_user_if_input (vm=0x7fffb5f5bf9c,
vum=0x7ffff7882a40 <vhost_user_main>, vui=0x7fffb65570c4, qid=0,
node=0x7fffb6577dac, mode=VNET_HW_INTERFACE_RX_MODE_POLLING)
at /home/sluong/vpp-master/vpp/build-data/../src/vnet/devices/virtio/vhost-user.c:1610
1610 bi_current = (vum->cpus[thread_index].rx_buffers)
[vum->cpus[thread_index].rx_buffers_len];
(gdb) p vum->cpus[thread_index].rx_buffers_len
$2 = 793212607
(gdb)
Apparently, some code accidentally wrote the bad value in rx_buffers_len.
rx_buffers_len should never be greater than 1024 since that is how many buffers
we request each time.
After debugging many hours, I discovered that the memory corruption happens
in the tx code path right here on line 2176.
{
vhost_copy_t *cpy = &vum->cpus[thread_index].copy[copy_len];
copy_len++;
cpy->len = bytes_left;
cpy->len = (cpy->len > buffer_len) ? buffer_len : cpy->len;
cpy->dst = buffer_map_addr;
cpy->src = (uword) vlib_buffer_get_current (current_b0) +
current_b0->current_length - bytes_left;
(gdb) p cpy
$3 = (vhost_copy_t *) 0x7fffb554077c
(gdb) p copy_len
$4 = 1025
(gdb) p &vum->cpus[3].rx_buffers_len
$8 = (u32 *) 0x7fffb5540784
copy_len is picking up the index entry 1024 before it was incremented. copy array has only
1024 members (0 - 1023 are valid).
The assignment here in cpy surely causes memory corruption. It is only discovered later
when the memory location that it corrupted is used.
The condition for the crash is to transmit jumbo frames under heavy volume. Since ring
size is 1024, with one packet taking up one index for frame size (less 2048), it does
not cause overflow. With jumbo frames, it requires multiple indices for one packet,
it can cause the overflow under heavy traffic.
The fix is to do copy out when we have 1000 entries in the array to avoid
overflow.
Change-Id: Iefbc739b8e80470f1cf13123113f8331ffcd0eb2
Signed-off-by: Steven <sluong@cisco.com>
|
|
(VPP-916)
The syntax for debug vhost-user is
debug vhost-user <on | off>
However, currently the code does not reject the invalid command such as below
debug vhost-user
debug vhost-user on blah
debug vhost-user off blah
The fix is to enforece the correct syntax and reject the command when invalid
option is entered.
Change-Id: I1a04ae8ddb6dd299aa6d15b043362964e685ddde
Signed-off-by: Steven <sluong@cisco.com>
|
|
Add runtime debug vhost-user on | off CLI to facilitate troubleshooting.
This feature is needed to avoid recompiling the code to debug vhost issues.
The debugging messages should not be on the data path to avoid performance
hit.
Change-Id: I4c40f65dbb222557cba3fb8706fa3b7b62eec95f
Signed-off-by: Steven <sluong@cisco.com>
|
|
and add adaptive mode support to receive queue
- Migrate vhost to use device infra which does the interface/queue to worker
thread assignment.
- Retire vhost thread CLI and corresponding code which assigns interface/queue
to worker thread. set interface placement should be used instead to customize
the interface/queue to worker thread assignment.
- Retire vhost interrupt/polling option when creating vhost-user interface.
Instead, set interface rx-mode should be used.
- Add code in vnet_device_input_unassign_thread to change the node state
to interrupt if the last polling interface has left the worker thread for the
device of the corresponding interface/queue.
- Add adaptive mode support. The node state is set to interrupt initially.
When the scheduler detects a burst of traffic, it switches the input node to
polling. Then we inform the device that we don't need interrupt notification.
When the traffic subsides, the scheduler switches the input node back to
interrupt. Then we immediately tell the driver that we want interrupt
notification again.
- Remove some duplicate code in vlib/main.c
Change-Id: Id19bb1b9e50e6521c6464f470f5825c26924d3a8
Signed-off-by: Steven <sluong@cisco.com>
|
|
When processing a vring descriptor which is outside of mmap, we disable
the interface and spit a message to shut/no shut the interface. This is
not practical as application using vhost cannot constantly checking the
logs and do the recovery. The proposed fix is to log an error, like
other errors that we encounter.
The other bug is buffer leak in the function rewind. At the end of the
while loop when b_current != b_head, we still have to give back 1 more
buffer or add 1 to rx_buffers_len.
Change-Id: I68c0b24f070e644cd8878f42272a7b518f14393f
Signed-off-by: Steven <sluong@cisco.com>
|
|
When the descriptor is chained via multiple parts, vhost is supposed
to reassemble the different parts to form a packet prior to passing
the packet to the next input node. However, bad packet was seen, having
bad ethertype, source, and destination mac addresses.
The problem was due to the destination pointer not being incremented as
each chain is processed. THe result was the first chain is copied to the
beginning of the buffer, the next chain is copied, then the last chain
is also copied to the beginning of the buffer. As a result, the ethertype,
source and destination mac, etc, are being overwritten by the very last
chain of the descriptor.
Change-Id: I78f9a91de68c85574047912576dcc311d7597e21
Signed-off-by: Steven <sluong@cisco.com>
|
|
Change-Id: I9664214652229b663c3e3ba7406b4ede96bfb123
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
|
When creating or modifying a vhost interface, verify if the path name
already existed and reject the command.
Change-Id: I8b2d33b77c847f774492874f7d194fa72c488479
Signed-off-by: Steven <sluong@cisco.com>
|
|
I had a bug where a requested size of 1G was resulting in
an aligned size of '1G + 2M', resulting in an OOM error.
Previous code was adding one huge page size
when memory is already aligned.
Change-Id: Idd3aa0e9b893fb3efccba6ae1c7161e26d3f9456
Signed-off-by: Pierre Pfister <ppfister@cisco.com>
|
|
Patch 6347 removed the socket file when the interface is deleted and
when VPP process is exitting. The CLI for deleting the interface has
builtin vlib_worker_thread_barrier_sync to prevent the worker
threads from running. Unfortunately, the CLI quit does not have the
builtin vlib_worker_thread_barrier_sync. As a result, it may cause the
worker thread to crash.
The fix is to add the vlib_worker_thread_barrier_sync in vhost_user_exit.
Change-Id: I1eff81170e131098f1799662f0ab48d6fca3def7
Signed-off-by: Steven <sluong@cisco.com>
|
|
- Unlink the file created for the socket when deleting vhost interface
if we are the server mode.
- Remove all vhost interfaces when VPP process is exitting.
Change-Id: Id9b676cd027bbd67b473bbd01901d1ecc4d8e6cb
Signed-off-by: Steven <sluong@cisco.com>
|
|
- Add cpu index to the vring structure for quick lookup
- Reduce the code that needs to be protected by
vlib_worker_thread_barrier_sync
- Set minimum timer no less than 1 ms
Change-Id: Iafef4bf6879a8efb350abf4e0f517e38f7ff7a8b
Signed-off-by: Steven <sluong@cisco.com>
|
|
This patch deprecates stack-based thread identification,
Also removes requirement that thread stacks are adjacent.
Finally, possibly annoying for some folks, it renames
all occurences of cpu_index and cpu_number with thread
index. Using word "cpu" is misleading here as thread can
be migrated ti different CPU, and also it is not related
to linux cpu index.
Change-Id: I68cdaf661e701d2336fc953dcb9978d10a70f7c1
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
|
(VPP-668)
When the number of open files is reached in the system, vhost may
encounter a failure in socket call and return from vhost-user-process.
The return terminates all attempts of incoming socket connections
in the future, even if the condition is reconciled.
The fix is to not return from vhost-user-process, record the error in
the interface, spit out the error, and retry the connection every 3 seconds.
Change-Id: I806baedf13e8c9b73e7c7820c094240f39949950
Signed-off-by: Steven <sluong@cisco.com>
|
|
To line up with "show interface placement," recently added. Otherwise,
"show int" refers only to "show interface placement," which tends to
annoy the cash customers...
Change-Id: Iea9e3681aeb051e2b0e1ecbf06706d98af9a3abf
Signed-off-by: Dave Barach <dave@barachs.net>
|
|
vhost currently supports only polling mode. This patch is to add
interrupt mode. When the interface is configured for interrupt
mode, our input node does not get called unless there is a packet
in the vring.
If a particular CPU has one interface configured for polling mode
and another in interrupt, the input node is set to polling for
that CPU.
This diffs also includes two crashes in vlib's dispatch_node. One is
included in https://gerrit.fd.io/r/#/c/5516. The other crash is in
the ASSERT. The ASSERT can become true when the caller of
dispatch_node is in a loop. The first call converted the node
to polling. The second call thereafter will hit the ASSERT.
Change-Id: If17b6d48b20d7d8605c6a161459828637173cd32
Signed-off-by: Steven <sluong@cisco.com>
|
|
When the VM is migrated, the driver sends VHOST_USER_VRING_GET_BASE
message to the device to get the vring offset. The device is
supposed to shut down the vring, and return the current vring offset.
What the code did was to shutdown the vring, initialize the vring,
and return 0 to the driver.
The fix is to first store last_avail_idx in the message and then close
the vring.
Change-Id: I432e9f50f36d89fe53a45e050edcf5e1218caf7a
Signed-off-by: Steven <sluong@cisco.com>
|
|
Clean up spurious binary API client link dependency on libvlib.so,
which managed to hide behind vlib_mains == 0 checks reached by
VLIB_xxx_FUNCTION macros.
Change-Id: I5df1f8ab07dca1944250e643ccf06e60a8462325
Signed-off-by: Dave Barach <dave@barachs.net>
|
|
Add operation_mode for create_vhost_user_if, modify_vhost_user_if,
and sw_interface_vhost_user_details. Only polling mode is supported
for these APIs. Other mode is rejected and gets
VNET_API_ERROR_UNIMPLEMENTED error.
Change-Id: I0596f4e2c087aa2b6f78eb3e0b63910b1859641e
Signed-off-by: Steven <sluong@cisco.com>
|
|
Change-Id: Ifac7d9134d03d79164ce6f06ae9413279bbaadb3
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
|
Change-Id: I1152db4b7d1602653d7d8b2c6cb28cf5c526c4ca
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
|
In the CLI parsing, below is a common pattern:
/* Get a line of input. */
if (!unformat_user (input, unformat_line_input, line_input))
return 0;
while (unformat_check_input (line_input) != UNFORMAT_END_OF_INPUT)
{
if (unformat (line_input, "x"))
x = 1;
:
else
return clib_error_return (0, "unknown input `%U'",
format_unformat_error, line_input);
}
unformat_free (line_input);
The 'else' returns if an unknown string is encountered. There a memory
leak because the 'unformat_free(line_input)' is not called. There is a
large number of instances of this pattern.
Replaced the previous pattern with:
/* Get a line of input. */
if (!unformat_user (input, unformat_line_input, line_input))
return 0;
while (unformat_check_input (line_input) != UNFORMAT_END_OF_INPUT)
{
if (unformat (line_input, "x"))
x = 1;
:
else
{
error = clib_error_return (0, "unknown input `%U'",
format_unformat_error, line_input);
goto done:
}
}
/* ...Remaining code... */
done:
unformat_free (line_input);
return error;
}
In multiple files, 'unformat_free (line_input);' was never called, so
there was a memory leak whether an invalid string was entered or not.
Also, there were multiple instance where:
error = clib_error_return (0, "unknown input `%U'",
format_unformat_error, line_input);
used 'input' as the last parameter instead of 'line_input'. The result
is that output did not contain the substring in error, instead just an
empty string. Fixed all of those as well.
There are a lot of file, and very mind numbing work, so tried to keep
it to a pattern to avoid mistakes.
Change-Id: I8902f0c32a47dd7fb3bb3471a89818571702f1d2
Signed-off-by: Billy McFall <bmcfall@redhat.com>
Signed-off-by: Dave Barach <dave@barachs.net>
|
|
Change-Id: I5b308eb39ae770d58d1498d7fafa49b236b3f534
Signed-off-by: Marek Gradzki <mgradzki@cisco.com>
|
|
This happens only on when compiled for older microarchitectures,
where BSF insutruction is used instead of TZCNT. BSF provides
undefined result if operand is 0.
Change-Id: I7a13350786a533428168595097ef01a560fde53b
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
|
Running trex in a VM with a bad config, trex sent a bogus pack from
the VM to the Virtual interface. It caused a crash.
Change-Id: I64d0197b444265553ab4c24f21e6a962e89cb587
Signed-off-by: Steven <sluong@cisco.com>
|
|
Change-Id: Ib0144ba3a9a09971d3946c932e8fed6d5c1ad278
Signed-off-by: Dave Barach <dave@barachs.net>
|
|
Vhost-user pool getting freed prematurely
Change-Id: I952821ec85efa68923d09a643c70b6b309ea2574
Signed-off-by: Wojciech Dec <wdec@cisco.com>
|
|
Change-Id: I69bbc447e1989adea40f052eac4550036b6e2e1e
Signed-off-by: Ole Troan <ot@cisco.com>
|
|
The unix connect() in vhost-user driver in VPP is blocking, and
a non-expedient accept() on the other side causes the entire VPP to hang.
Solution: set the nonblocking flag for the socket fd before calling
connect(), and set the socket back to blocking after the accept() succeeds.
Change-Id: Ia5ee782037eeffabdad71db8241f476a048a4f6f
Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
|
|
Fix for VPP-573.
Change-Id: If7d9690901efebf62fdf28219097153d98c79c0c
Signed-off-by: Wojciech Dec <wdec@cisco.com>
|
|
Change-Id: I7b51f88292e057c6443b12224486f2d0c9f8ae23
Signed-off-by: Damjan Marion <damarion@cisco.com>
|
Pierre Pfister
Steven
Damjan Marion
Dave Barach
Billy McFall
Marek Gradzki
Wojciech Dec
Ole Troan
Andrew Yourtchenko