| Age | Commit message (Collapse) | Author |
|---|
|
With an empty thread mask, thr_tbl[] in schedule_group_create() ends
up being zero size, which is undefined behavior. Fix by using a static
array size.
Fixes GCC undefined sanitizer error:
odp_schedule_basic.c:1922:6: runtime error: variable length array bound evaluates to non-positive value 0
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Update all implementation files to use the new shorter copyright format:
/* SPDX-License-Identifier: BSD-3-Clause
* Copyright (c) <YEAR> <COPYRIGHT HOLDER>
*/
Inherited DPDK copyright messages have also been updated to utilize SPDX
format and now list the original source files.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
|
|
Use _odp_strcpy() instead of strncpy().
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Rename the "print" internal module to "string". It contains the
_odp_snprint() function. Other string and printing related internal
functions can be added in this module in the future.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Reorganize some conditionals in schedule_loop_sleep() in order to make
them clearer, and to avoid calling odp_time_cmp() needlessly in some
situations.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
When using inline timers and power saving sleep, limit sleep duration
according to the next timeout.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Simplify linux-gen implementation by using the new add nsec
function instead of sum time stamps function.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Remove ODP_ prefix from implementation internal config defines to
distinguish them from official APIs.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
|
|
Fix potential integer overflows reported by Coverity.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Optionally, when there are no events to schedule, sleep for a while to
save power. Applies to all waiting schedule calls.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add power saving config file options poll_time and sleep_time.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add function schedule_fn_t.sched_api, with which each scheduler
returns its API function table. This way the upper level doesn't need
to know it at compile time.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add new configuration option 'sched_basic.order_stash_size', which can be
used to adjust the size of implementation internal per thread ordered queue
reorder stash. If 'order_stash_size' > 0, events may be dropped by the
implementation if the target queue is full. To prevent this set
'order_stash_size' to 0.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reported-and-tested-by: Carl Wallen <carl.wallen@nokia.com>
Reviewed-by: Jere Leppänen <jere.leppanen@nokia.com>
|
|
Change group mask variables to 64 bits in order to
increase maximum number of supported groups from 32 to 64.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-and-tested-by: Matias Elo <matias.elo@nokia.com>
|
|
Add inline header for fast path schedule API functions.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Prefix implementation internal debug macro names with underscore.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Use _odp_snprint() to format full lines for ODP_PRINT()
output. Full lines remain more readable than line fragments,
when other threads print simultaneously into the same log.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add new configuration file options for adjusting scheduler burst sizes per
scheduled queue type. Use 32-bit fast path variables for improved
arithmetic operation performance.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Remove an extra level of hierarchy from queue_entry_t, and make the
type cache aligned.
Signed-off-by: Jere Leppänen <jere.leppanen@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
|
|
Use a relaxed atomic load in schedule loop to document that
the data race between prio_grp_mask write and read is intentional.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
When packet input poll does not return packets, move to the
next spread instead of the next group/priority. Since packet
input queues are not removed from scheduling when they are
empty, this branch may be taken often. Scheduling is more
fair (to other queues in the same group/priority) when the
remaining spreads are not skipped.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Added mask of groups that have queues created on a priority
level. This allows scheduler to skip over priority levels that
are not relevant to any of those groups that the thread
belongs to.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Replace grp_weight table with modulo of the group index. This uses
less thread local memory (TLS) and round-robins indexes evenly.
Modulo calculation should not hurt performance (of an out-of-order
CPU) as its result is used on the next schedule call.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Modify scheduler to be more strict on priority. Check the
highest priority level on all groups before moving to the
next priority level. This avoids priority inversion when
groups do not contain queues from the same priority levels.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Make it easier to follow when scheduler moves to the next
spread, group or priority. Flatten packet input branch and
change a break to a return inside it. Improve documentation of
the brances within the loop.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Moved schedule priority loop outside of do_schedule_grp()
function. Remaining code in do_schedule_grp() is not
changed, just re-aligned after removal of the for-loop.
This enables later changes to group and priority
selection algorithm.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Changed 'unsigned int' to uint32_t as those are in practice
the same type, but uint32_t is well-defined and shorter type
name.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Janne Peltonen <janne.peltonen@nokia.com>
|
|
Implement odp_schedule_order_wait() by using the existing order_lock()
(not to be confused with ordered locks) function that does what we need
and does not require unlock. Support is added for the basic and scalable
scheduler but not for the SP scheduler where the function is a nop.
Signed-off-by: Janne Peltonen <janne.peltonen@nokia.com>
Reviewed-by: Jerin Jacob <jerinj@marvell.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Add _ODP_ prefixes to implementation internal helper macros and collect
common macros into a single header file (odp_macros_internal.h). Unused
macros (DIV_ROUND_UP, odp_container_of) have been removed.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Tuomas Taipale <tuomas.taipale@nokia.com>
|
|
Random data table had same values multiple times. Update table
with shuffled values between 0 and 127. This way random data
has even distribution of values, which results even distribution
of spread modulos.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Use common _odp_event_hdr_t in odp_buffer_hdr_t. Cache line alignment
constraint has been moved to odp_buffer_hdr_t.
The implementation internal scheduler and queue interfaces have been
modified to use event headers instead of buffer headers.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Add scheduler group names to the output of odp_schedule_print() function to
provide additional debug information.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Fix "invalid use of structure with flexible array member" errors when
building with 'pedantic' option. Internal queue_num_tbl_t struct was not
used, so it has been removed.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Fix "ISO C does not allow extra ';' outside of a function" errors when
building with 'pedantic' option.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
Make implementation internal global data SHM block naming consistent
(_odp_*_global).
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
When there are multiple spreads with the minimum number of
queues, select the spread randomly among those. This makes spreads
more evenly loaded, especially when each group/prio has only few
queues (e.g. many groups/prios, but only 1-2 queues per group/prio).
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Update number of threads per spread statistics on group create and
destroy. A thread can be placed to a group with create or join
calls. Dynamic scheduler uses thread counts and did not work
correctly, if threads were added to a group with the create call.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Added scheduler group and spread to debug print output. Spread is printed
only with basic scheduler (SP does not have it). Changed queue debug
print output to match odp_schedule_print() output, which prints scheduler
internal priority level (0 == highest prio).
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Current, queue index based spread allocation causes dependency
to queue create order and spread. Also, it does not take account
group/prio, which allows e.g. all high prio queues to end up into
a single spread.
Change initial spread selection to be dynamic, so that spreading
works more optimally also when dynamic load balance is disabled.
When queue is created, select the spread that has least queues
at that moment. Since queues may be created before threads are
allocated into groups, use just number of queues (instead of queues
divided by threads) as measurement of the current load.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Change all scheduler locks to ticket locks. Ticket lock may
give better performance on modern CPUs. It is also more fair,
as threads acquire the lock in the order they got their tickets -
one or few threads cannot acquire the same lock many times in a row.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Change global mask_lock to per group mask_locks. This lock protects
priority queue mask and count, which are per group objects. Also,
separate locks lower contention on update_queue_count(), which is
called from the fast path (odp_schedule()), when load is not
in balance.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Estimate current load on a spread ring as number of queues
allocated to it divided by number of threads preferring the
spread. Each thread runs a series of spread load balance
checks starting at every 1M schedule calls.
This helps especially when there are less threads in a group
than the number of spreads configured. Scheduler moves more
queues to those spread rings that have more thread capacity.
This works best when all threads and queues are equally active.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Implemented the new debug print function for all three schedulers.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Jere Leppänen <jere.leppanen@nokia.com>
|
|
For better code readability, rename spread index variables
to spr or spread.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add group level to queue count and queue mask data structures.
This tells exactly if queues have been placed to a particular
scheduler ring (priority queue).
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Add statistics that count the number of threads preferring
a spread (of a group).
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Thread mask and schedule group info may be requested also on
prefined schedule groups.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
In process mode the value of_odp_schedule_configured was not synchronized
to forked processes.
Signed-off-by: Matias Elo <matias.elo@nokia.com>
Reviewed-by: Petri Savolainen <petri.savolainen@nokia.com>
|
|
When prio_spread is not a power of two, ring size is rounded up
which causes the assert to fail. Limit max number of queues
instead of the assert.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
|
|
Each ring can store in maximum ring size minus one queues. Move
ring mask initialization into scheduler global init.
Signed-off-by: Petri Savolainen <petri.savolainen@nokia.com>
Reviewed-by: Matias Elo <matias.elo@nokia.com>
Tested-by: Shijith Thotton <sthotton@marvell.com>
|
