summaryrefslogtreecommitdiffstats
path: root/test/framework.py
AgeCommit message (Collapse)AuthorFilesLines
2022-02-24tests: better reporting for unexpected packetsKlement Sekera1-4/+2
Raise a new UnexpectedPacketErrror, when a packet is captured unexpectedly. This pretty-prints a terse description of said packet. Type: improvement Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: Ibac19fc5bbd82a150fec3c90940a37af6344fd4f
2022-02-21tests: add enhanced packet counter verificationKlement Sekera1-12/+74
Add support for inline packet counter verification to send_and_* functions. Diff dictionary is a dictionary of dictionaries of interesting stats: diff_dictionary = { "err" : { '/error/counter1' : 4, }, sw_if_index1 : { '/stat/segment/counter1' : 5, '/stat/segment/counter2' : 6, }, sw_if_index2 : { '/stat/segment/counter1' : 7, }, } It describes a per sw-if-index diffset, where each key is stat segment path and value is the expected change for that counter for sw-if-index. Special case string "err" is used for error counters. This then allows more precise packet counter verification by first defining a "zero" dictionary, e.g. for ED NAT: cls.no_diff = StatsDiff({ pg.sw_if_index: { '/nat44-ed/in2out/fastpath/tcp': 0, '/nat44-ed/in2out/fastpath/udp': 0, '/nat44-ed/in2out/fastpath/icmp': 0, '/nat44-ed/in2out/fastpath/drops': 0, '/nat44-ed/in2out/slowpath/tcp': 0, '/nat44-ed/in2out/slowpath/udp': 0, '/nat44-ed/in2out/slowpath/icmp': 0, '/nat44-ed/in2out/slowpath/drops': 0, '/nat44-ed/in2out/fastpath/tcp': 0, '/nat44-ed/in2out/fastpath/udp': 0, '/nat44-ed/in2out/fastpath/icmp': 0, '/nat44-ed/in2out/fastpath/drops': 0, '/nat44-ed/in2out/slowpath/tcp': 0, '/nat44-ed/in2out/slowpath/udp': 0, '/nat44-ed/in2out/slowpath/icmp': 0, '/nat44-ed/in2out/slowpath/drops': 0, } for pg in cls.pg_interfaces }) and then to specify only changed counters directly when calling one of send_and_* functions: self.send_and_assert_no_replies( self.pg0, pkts, msg="i2o pkts", stats_diff=self.no_diff | { "err": { '/err/nat44-ed-in2out-slowpath/out of ports': len(pkts), }, self.pg0.sw_if_index: { '/nat44-ed/in2out/slowpath/drops': len(pkts), }, } ) operator | is overloaded by StatsDiff class to perform a deep merge operation, so in above case, dictionaries for "err" and self.pg0.sw_if_index do not overwrite whole sub-dictionaries, rather the contents are merged, assuring that all the remaining counters are verified to be zero. Type: improvement Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I2b87f7bd58a7d4b34ee72344e2f871b2f372e2d9
2022-02-21tests: add a generalised counter assert functionKlement Sekera1-2/+11
Type: improvement Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I5e0fd1019372df0cd403725b8cac52363af13718
2022-02-21tests: allow not removing vpp objects after testKlement Sekera1-1/+3
Some tests might want to keep vpp config in between test functions. Add a flag to allow that. Type: improvement Signed-off-by: Klement Sekera <klement.sekera@gmail.com> Change-Id: I90243cd667dce922b43b381f3d52f4ac0f6bf3a7
2022-02-18tests: Consolidate the implementations of send_and_expect_XNeale Ranns1-0/+12
Type: test there were multiple instances of send_and_expect_load_balancing and a send_and_expect_one_itf which has the same functionality as send_and_expect_one. Put one implementation of both in framework.py (where the other send_and_X functions reside). Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: I7f629d440220bee29368067f475059322e1134f7
2022-02-17tests: make tests less make dependentKlement Sekera1-100/+49
Implement command line argument parsing instead of passing arguments via environment variables. Add script for running tests without having to invoke make. Deprecate running tests via make. Type: improvement Change-Id: I2e3054a61a2ae25d460e9be00be7d7705fbf943e Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2022-02-05tests: mark the test suites broken when ASan is enabledNaveen Joy1-0/+16
Mark broken test suites using @tag_fixme_asan.The main issue is that some tests do not pass with ASan. These will need to be fixed, but it takes time and in the meantime, new issues are introduced that trip up ASan. When this tag is set and Asan is enabled, failing ASan tests are skipped, so incremental fixes can pass the CI Type: improvement Change-Id: I02602eb74234c25a4c701279e14704b81d4c5b71 Signed-off-by: Naveen Joy <najoy@cisco.com>
2022-01-24nat: TCP state tracking based on RFC 7857/RFC 6146Klement Sekera1-2/+2
Implement proper state machine based on above RFCs. ACKs to SYNs/FINs are no longer required/tracked. This is more friendly to peers and accounts for lost packets and retransmits. This change also means that all traffic is translated and forwarded while in transitory timeout, which helps delivering e.g. retransmitted FINs, FINACKs and other messages. Also support reopening a session in transitory timeout after seeing both FINs by seeing both SYNs again. This helps quick connection reestablishment if the peers want to. Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Signed-off-by: Miklos Tirpak <miklos.tirpak@gmail.com> Change-Id: Ibf521c79463472db97e593bfa02b32b4a06dfd2a
2021-12-10ip: reassembly: handle atomic fragments correctlyKlement Sekera1-1/+4
If a fragment arrives with fragment offset = 0 and M = 0, it means that this is actually a complete packet and per RFC 8200, it should be treated independently from other fragments. This patch does that. Fragmentation header is stripped and fragment is forwarded irregardles of other existing reassemblies in case of full reassembly and treated the same way as regular packet in shallow virtual reassembly. Type: improvement Change-Id: If3322d5e3160cd755b8465a642702a9166d46cc2 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-11-22vlib: add virtual time supportBenoît Ganne1-0/+4
Type: feature Change-Id: Iabd76558e9c72ed8286cfeeb1fbaa4fde4832a90 Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-11-02build: remove unnecessary executable bitsRay Kinsella1-0/+0
Run 'find src -executable -type f', remove unnecessary executable bits from the source tree. Type: refactor Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I070e22c8fb2ef8712bc3ea620727ee5ab3d9a9fb
2021-10-14nat: static mappings in flow hashKlement Sekera1-0/+2
Put static mappings in flow hash, drop existing hash tables used for static mappings. Drop refcount variables and use hash table as a single point of truth. Allow creating a static mapping conflicting with dynamic mapping, which will take precedence after dynamic mapping is freed, so that the existing flow can finish transferring data. Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Idfde8efabc09971be38921d4b0ca5ccf4e9fe412
2021-10-12Revert "nat: static mappings in flow hash"Ole Troan1-2/+0
This reverts commit 69b7599e4b061a8996205f0304232ede84cb70d4. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: If531b122ae5a9f91c2fe6eaa0da69922a91f16d3
2021-10-12nat: static mappings in flow hashKlement Sekera1-0/+2
Put static mappings in flow hash, drop existing hash tables used for static mappings. Drop refcount variables and use hash table as a single point of truth. Allow creating a static mapping conflicting with dynamic mapping, which will take precedence after dynamic mapping is freed, so that the existing flow can finish transferring data. Type: fix Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Ieeba691d83a83887d0a0baccd5f3832f66126096
2021-10-05build: don't hardcode triplet, allow specifying custom lib dirDamjan Marion1-16/+4
Type: fix Change-Id: I33f364fda88914f88f9b976cb83e6d3ff466f0bb Signed-off-by: Damjan Marion <damarion@cisco.com> Signed-off-by: Ray Kinsella <mdr@ashroe.eu>
2021-09-27classify: Large and nested classifer unit testsRay Kinsella1-8/+16
Type: test Large and nested unit tests to test AVX-512 optimized versions of the classify hash and match algorithims. Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: Ie423fee5e0fd1cb4bdf3bec8e0230a5f7cfc75fc
2021-06-16api: remove custom dumpFilip Tehlar1-2/+0
Type: improvement Change-Id: I4b9b2be8817be10e46accc19219deb2b544f266b Signed-off-by: Filip Tehlar <ftehlar@cisco.com>
2021-06-14pg: A Tunnel mode variant of a pg interfaceNeale Ranns1-2/+28
Type: feature this allows VPP to simulate linux tun devices. Signed-off-by: Neale Ranns <neale@graphiant.com> Change-Id: I3adf38b49a254804370f78edd5d275d192fd00a6
2021-05-21tests: add role to VCLAppWorkerDave Wallace1-2/+3
- Improve readability of log output with respect to worker role Type: test Signed-off-by: Dave Wallace <dwallacelf@gmail.com> Change-Id: I1a5787421b31483fa1a2558951568417dfc111d9
2021-05-20tests: don't buffer worker stdin/stdoutDave Wallace1-2/+3
- Required to get VCLAppWorker stdin / stdout data into log.txt when a test fails. Type: test Change-Id: I4db467986940f701750aec6d842a4d9aee22b4e6 Signed-off-by: Dave Wallace <dwallacelf@gmail.com>
2021-05-04stats: adding symlinks for nodes and interfaces in the stat segmentArthur de Kerhor1-1/+3
A given interface counter (e.g rx) can be accessed via /interfaces/<interface_name>/<counter_name>. Same goes with nodes: /nodes/<node_name>/<counter_name> As interfaces may contain '/' in their names, these are replaced by '_' in symlinks Also added 2 tests for the stat segment Type: feature Signed-off-by: Arthur de Kerhor <arthurdekerhor@gmail.com> Change-Id: I27da252e7b3dc177815616ca46b5c200a456bf0f Signed-off-by: Ole Troan <ot@cisco.com>
2021-04-28tests: add worker count to class descriptionKlement Sekera1-1/+9
While running test: ============================================================================== NAT44ED Test Case ============================================================================== ==> ============================================================================== NAT44ED Test Case [main thread only] ============================================================================== ============================================================================== NAT44ED Test Case [1 worker thread] ============================================================================== ============================================================================== NAT44ED Test Case [4 worker threads] ============================================================================== Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: I61904b411ff1b58700564698079dc4c07cea5c5e
2021-04-16tests: cpus awarenessKlement Sekera1-60/+75
Introduce MAX_CPUS parameters to control maximum number of CPUs used by VPP(s) during testing, with default value 'auto' corresponding to all CPUs available. Calculate test CPU requirements by taking into account the number of workers, so a test requires 1 (main thread) + # of worker CPUs. When running tests, keep track of both running test jobs (controlled by TEST_JOBS parameter) and free CPUs. This then causes two limits in the system - to not exceed number of jobs in parallel but also to not exceed number of CPUs available. Skip tests which require more CPUs than are available in system (or more than MAX_CPUS) and print a warning message. Type: improvement Change-Id: Ib8fda54e4c6a36179d64160bb87fbd3a0011762d Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-12tests: support attaching to existing vppKlement Sekera1-29/+70
Introduce a new option DEBUG=attach to run a test against existing already running vpp. A new target 'make test-start-gdb' will spawn VPP in gdb for this purpose. Customization options explained in test-help. Type: improvement Change-Id: Ia160a85b33da3b2df292d44bb95729af9dd9da96 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-04-09tests: improve namingKlement Sekera1-12/+12
Change capture to pcap to remove confusion in framework.py Type: improvement Signed-off-by: Klement Sekera <ksekera@cisco.com> Change-Id: Idf8bd61f174b8b2df3bbd13908e54d597da4a83b
2021-04-01tests: support injecting multiple worker pcaps on one PGKlement Sekera1-4/+11
This change allows one to inject multiple streams for different workers on the same PG interface at the same time. Type: improvement Change-Id: I29d80369aabada261eda466e5a5d8d3518bb8bc8 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-25stats: python vpp_stats rewrite to access stat segment directlyOle Troan1-1/+1
This module implement Python access to the VPP statistics segment. It accesses the data structures directly in shared memory. VPP uses optimistic locking, so data structures may change underneath us while we are reading. Data is copied out and it's important to spend as little time as possible "holding the lock". Counters are stored in VPP as a two dimensional array. Index by thread and index (typically sw_if_index). Simple counters count only packets, Combined counters count packets and octets. Counters can be accessed in either dimension. stat['/if/rx'] - returns 2D lists stat['/if/rx'][0] - returns counters for all interfaces for thread 0 stat['/if/rx'][0][1] - returns counter for interface 1 on thread 0 stat['/if/rx'][0][1]['packets'] - returns the packet counter for interface 1 on thread 0 stat['/if/rx'][:, 1] - returns the counters for interface 1 on all threads stat['/if/rx'][:, 1].packets() - returns the packet counters for interface 1 on all threads stat['/if/rx'][:, 1].sum_packets() - returns the sum of packet counters for interface 1 on all threads stat['/if/rx-miss'][:, 1].sum() - returns the sum of packet counters for interface 1 on all threads for simple counters Type: refactor Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1fe7f7c7d11378d06be8276db5e1900ecdb8f515 Signed-off-by: Ole Troan <ot@cisco.com>
2021-03-20tests: add support for worker awarenessKlement Sekera1-25/+33
VppTestCase now has vpp_worker_count property set to number of workers. This can be overriden by child classes. Also overriden by VPP_WORKER_CONFIG variable for legacy reasons. Type: improvement Change-Id: Ic328bacb9003ddf9e92815767653bd362aa7f086 Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-15tests: use socket transport instead of shared memoryOle Troan1-12/+16
Type: improvement Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I9e65c94a5a05047a5104e9361ea36eac77b40442 Signed-off-by: Ole Troan <ot@cisco.com>
2021-03-15tests: fix NoneType printing if VPP died earlyKlement Sekera1-12/+15
Make error message more meaningful. Type: fix Change-Id: I3c49cb179c3ee7a59657b3ae9a06311f57dc52ac Signed-off-by: Klement Sekera <ksekera@cisco.com>
2021-03-02classify: fix multiple filters supportBenoît Ganne1-7/+9
This fix the classify filter if we attach several different filters. This also fix some issues with l3 and l4 parsing. Type: fix Change-Id: I9dc6c55049a3bbc0110d1097b40d9da27633626b Signed-off-by: Benoît Ganne <bganne@cisco.com>
2021-02-15tests: explicitly close subprocess filesPaul Vinciguerra1-0/+2
Resolve the following error: /vpp/test/framework.py:657: ResourceWarning: unclosed file <_io.BufferedReader name=6> del cls.vpp Object allocated at (most recent call last): File "/usr/lib/python3.8/subprocess.py", lineno 844 self.stdout = io.open(c2pread, 'rb', bufsize) Type: test Change-Id: Ia2974da594f0582dcff1f4bdf40d25475769c46c Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2021-02-08tests: allow for externally supplied VPP workers config for testsAndrew Yourtchenko1-1/+17
Allow to supply the external VPP worker config for tests which do not specify the workers config explicitly, and use the tags infra to flag those that need attention in this configuration. This commit shows one example use of such a tag, there will be a separate commit with the rest of the places needing it, since that change is rather mechanical. Thus, the assumption is that the test should by default be agnostic of the VPP configuration, unless it explicitly specifies so. Type: test Change-Id: I3c0077e4e22a75cb9561fb98d3b783b93486b2be Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2021-01-22tests: add generalized tags for tests, use them for run-solo testsAndrew Yourtchenko1-3/+29
We have accumulated several scenarios in prod or wishlists where it would be useful to have a general infra to say yes/no about a certain test, and potentially make decisions based on that, for example: - runs solo (aka 'time-dependent') - (wishlist) part of quick smoke-test set - (wishlist) intermittent failure unrelated to timing - (wishlist) test broken with a multi-worker config in vpp Refactor the current "run-solo" code to allow for this extension. Type: test Change-Id: Ia5b3810e57c0543753c8e0dc4dc0cfb4a30b36ac Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com> Signed-off-by: Klement Sekera <ksekera@cisco.com>
2020-12-08tests: restore vapi C++ test under Centos-8Paul Vinciguerra1-8/+0
The Centos-7 compiler had problems with vapi_cpp_test.cpp. The Centos-8 compiler does not. This change adds back the test. ============================================================================== VAPI test ============================================================================== run C VAPI tests 1.55 OK run C++ VAPI tests 0.44 OK ============================================================================== TEST RESULTS: Scheduled tests: 2 Executed tests: 2 Passed tests: 2 ============================================================================== Type: test Change-Id: Ic6a655ae99eb3ba0e73b7feef4436a0ca5fd92be Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-12-06tests: remove the dependency on subprocess32Paul Vinciguerra1-13/+1
Type: test Change-Id: I39afe56a085648d881a18c7eafcf7caf2f375f31 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-09-22lisp: Move to pluginNeale Ranns1-0/+2
Type: refactor Change-Id: I54df533a8f863c4e49742903cf2457f18b4fc506 Signed-off-by: Neale Ranns <nranns@cisco.com>
2020-08-27tests: "force solo" testcase supportAndrew Yourtchenko1-1/+16
Some of the tests are time-sensitive, and at present require a non-trivial modification in order to run at high concurrency. Without these modifications, they intermittently fail, and require the test retries. Rather than setting them to the extended tests and forgetting about them, put them into a "solo" set, which gets run in a single-threaded mode after the rest of the tests are done. Mark a few of the tests that showed errors during TEST_JOBS=48 as forced-solo. Also, give a better diagnostic if the testcase misses a docstring needed to represent it in the diagnostic outputs. Type: fix Change-Id: I33fe62eb17edc1885bd2c3523892051d52da6546 Signed-off-by: Andrew Yourtchenko <ayourtch@gmail.com>
2020-05-05tests: clean up loggingPaul Vinciguerra1-12/+17
Tests currently expect the logger to be poked from run_tests.py. The tests should run without any magic values. This change sets a default null logger and removes the hasattr checks for the logger. For reference, see: https://docs.python-guide.org/writing/logging/ Type: test Change-Id: I98f953d73d12d00e74b59c94a0fb8c7a625b9c44 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-05-04misc: binary api fuzz test fixesDave Barach1-0/+5
Add a hook to src/vlibapi/api_shared.c to fuzz (screw up) binary API messages, e.g. by xoring random data into them before processing. We specifically exempt client connection messages, and inband debug CLI messages. We step over msg_id, client index, client context, and sw_if_index. Otherwise, "make test" vectors fail too rapidly to learn anything. The goal is to reduce the number of crashes caused to zero. We're fairly close with this patch. Add vl_msg_api_max_length(void *mp), which returns the maximum plausible length for a binary API message. Use it to hardern vl_api_from_api_to_new_vec(...) which takes an additional argument - message pointer - so it can verify that astr->length is sane. If it's not sane, return a u8 *vector of the form "insane astr->length nnnn\0". Verify array lengths in vl_api_dhcp6_send_client_message_t_handler(...) and vl_api_dhcp6_pd_send_client_message_t_handler(...). Add a fairly effective binary API fuzz hook to the unittest plugin, and modify the "make test" framework.py to pass "api-fuzz { on|off }" to enable API fuzzing: "make API_FUZZ=on TEST=xxx test-debug" or similar Type: improvement Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I0157267652a163c01553d5267620f719cc6c3bde
2020-04-28tests: implement ipaddress convenience methodsPaul Vinciguerra1-1/+4
Add vpp specific properties to ip addresses for use in the api. .vapi_af -- returns [ADDRESS_IP4, ADDRESS_IP6] .vapi_af_name -- returns the string ['ip4', 'ip6'] Update tests to demonstrate usage. Type: feature Change-Id: I43447a1522769d99f89debdc714c51700068d771 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-04-28vlib: startup multi-arch variant configurationRay Kinsella1-0/+8
Support for startup node multi-arch variant selection through startup.conf. This is to facilitate unit, functional testing and benchmarking of non-default multi-arch variant node code path. Also added parameters to make test, to specific using multi-arch variants in unit testing. Type: improvement Signed-off-by: Ray Kinsella <mdr@ashroe.eu> Change-Id: I94fd332bb629683b7a7dd770ee9f615a9a424060
2020-03-12vppinfra: refactor clib_timebase_tDave Barach1-1/+6
Add a clib_time_t * argument to clib_timebase_init(...), to encourage client code to share the vlib_main_t's clib_time_t object. Display the current day / date in GMT via the "show time" debug CLI. Fix the test framework so it processes the new "show time" output format. Type: refactor Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: I5e52d57eb164b7cdb6355362d520df6928491711
2020-03-10tests: add running_gcov_tests to framework.pyDave Barach1-0/+7
Mark a few code coverage tests appropriately: @unittest.skipUnless(running_gcov_tests, "part of code coverage tests") def test_my_coverage_unittest(self): <etc> Almost exactly like "make EXTENDED_TESTS=yes test". Type: feature Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Ic6d0a097a608ba93442b3d88252f66f8e6805b97
2020-02-07tests: support python 3.8Ole Troan1-2/+1
Make test framework python3 version independence. Type: fix Signed-off-by: Ole Troan <ot@cisco.com> Change-Id: I1ef1eb77b6c1f422ebc4dad0818f87c8e587b34b
2020-01-14tests: fix worker thread initializationPaul Vinciguerra1-4/+4
from threading.thread __init__: This constructor should always be called with keyword arguments. If a subclass overrides the constructor, it must make sure to invoke the base class constructor (Thread.__init__()) before doing anything else to the thread. Type: test Change-Id: Ifa89202e97053a4baf19e9a0ca0913430d5087a3 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2020-01-02tests: configure 32 mb physmemDave Barach1-0/+1
Absolutely nothing good happens when we force the kernel to briefly map and then unmap 16gb as vpp starts. Effect exacerbated when TEST_JOBS = 20...40, and so forth. Type: test Signed-off-by: Dave Barach <dave@barachs.net> Change-Id: Id8e3ce1763cad3a0891d5d6c8d2c1e3e610682d7
2019-12-23ipsec: Test and fix IPSec worker hand-offNeale Ranns1-4/+4
Type: fix Change-Id: I5cb9a3845ddbc5f4de4eb4e9c481f606fe5cec9a Signed-off-by: Neale Ranns <nranns@cisco.com>
2019-12-20tests: don't prompt to launch gdb for sanity test casePaul Vinciguerra1-10/+18
Type: test Change-Id: I4c54121b76b341381a819cee928c3c2455a83503 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>
2019-12-19papi: lazily initialize stats clientPaul Vinciguerra1-14/+0
remove wait-loop on stats socket from test framework. Type: refactor Change-Id: I5bb95a7c597707a87f9d9a471215c4b4af1a2280 Signed-off-by: Paul Vinciguerra <pvinci@vinciconsulting.com>