c - 是不是内核卡住

标签 c linux-kernel

我们在具有以 1.25GHz 运行的多核的 linux 嵌入式系统中缺少中断。

背景:

  • 内核版本:2.6.32.27
  • 我们有需要实时性能的用户空间进程。
  • 它们在 1 毫秒的边界内运行。
    • 也就是说他们有望在 1ms 内完成一组任务，最多可能需要 800uS 左右。
  • 我们有一个外部组件 FPGA，它通过配置为边沿触发中断的 GPIO 引脚向多核处理器提供 1 毫秒和 10 毫秒的中断。
  • 这些中断在内核驱动程序中处理。

软件架构是这样一种方式，即用户进程在完成其工作后将对 GPIO 驱动程序执行 ioctl。

在此 ioctl 中，驱动程序会将进程置于 wakeup_interruptible 状态。 每当收到下一个 1ms 中断时，ISR 将唤醒进程。重复此循环。

1ms 和 10ms 中断都使用 smp_affinity 路由到处理器的单个内核。

问题:

  • 有时我们会发现错过了一些中断。
    • (即 ISR 本身不会被调用)。
  • 在 12 到 20 毫秒后，ISR 被正常触发。
  • 我们可以通过分析连续 ISR 调用之间的持续时间，并让计数器在 ISR 中递增来理解这一点。

这主要发生在进程级别的高系统负载期间，并且是随机的并且难以重现。

我附上了骨架代码。

首先我必须确定是硬件问题还是软件问题。由于提供中断的是 FPGA，因此我们对硬件没有太多怀疑。

这个内核卡住了吗？这是最有可能的情况，因为 cpu 周期在增加。

会不会是由于热条件导致 CPU 卡住的情况？如果是这样，那么 cpu 周期就不会首先增加。

任何调试/隔离根本原因的指示都会有很大帮助 考虑到我们正在处理的内核版本以及分析/调试 此内核版本中可用的功能。

骨架代码:

/* Build time Configuration */

/* Macros */
DECLARE_WAIT_QUEUE_HEAD(wait);

/** Structure Definitions */
/** Global Variables */
gpio_dev_t gpio1msDev, gpio10msDev;
GpioIntProfileSectorData_t GpioSigProfileData[MAX_GPIO_INT_CONSUMERS];
GpioIntProfileSectorData_t *ProfilePtrSector;
GpioIntProfileData_t GpioProfileData;
GpioIntProfileData_t *GpioIntProfilePtr;
CurrentTickProfile_t TimeStamp;
uint64_t ModuleInitDone = 0, FirstTimePIDWrite = 0;
uint64_t PrevCycle = 0, NowCycle = 0;
volatile uint64_t TenMsFlag, OneMsFlag;
uint64_t OneMsCounter;
uint64_t OneMsIsrTime, TenMsIsrTime;
uint64_t OneMsCounter, OneMsTime, TenMsTime, SyncStarted;
uint64_t Prev = 0, Now = 0, DiffTen = 0, DiffOne, SesSyncHappened;
static spinlock_t GpioSyncLock = SPIN_LOCK_UNLOCKED;
static spinlock_t IoctlSyncLock = SPIN_LOCK_UNLOCKED;
uint64_t EventPresent[MAX_GPIO_INT_CONSUMERS];

GpioEvent_t CurrentEvent = KERN_NO_EVENT;
TickSyncSes_t *SyncSesPtr = NULL;


/** Function Declarations */

ssize_t write_pid(struct file *filep, const char __user * buf, size_t count, loff_t * ppos);
long Gpio_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg);

static const struct file_operations my_fops = {
 write:write_pid,
 compat_ioctl:Gpio_compat_ioctl,
};




/**
 * IOCTL function for GPIO interrupt module
 *
 * @return
 */
long Gpio_compat_ioctl(struct file *filep, unsigned int cmd, unsigned long arg)
{
int len = 1, status = 0;
    uint8_t Instance;
    uint64_t *EventPtr;
    GpioIntProfileSectorData_t *SectorProfilePtr, *DebugProfilePtr;
    GpioEvent_t EventToGive = KERN_NO_EVENT;
    pid_t CurrentPid = current->pid;

    spin_lock(&IoctlSyncLock);  // Take the spinlock
    Instance = GetSector(CurrentPid);
    SectorProfilePtr = &GpioSigProfileData[Instance];
    EventPtr = &EventPresent[Instance];
    spin_unlock(&IoctlSyncLock);

    if (Instance <= MAX_GPIO_INT_CONSUMERS)
    {
        switch (cmd)
        {
        case IOCTL_WAIT_ON_EVENT:
            if (*EventPtr)
            {
                /* Dont block here since this is a case where interrupt has happened
                 * before process calling the polling API */
                *EventPtr = 0;
                /* some profiling code */
            }
            else
            {
                status = wait_event_interruptible(wait, (*EventPtr == 1));
                *EventPtr = 0;
            }

            /* profiling code */

            TimeStamp.CurrentEvent = EventToGive;
            len = copy_to_user((char *)arg, (char *)&TimeStamp, sizeof(CurrentTickProfile_t));
            break;
        default:
            break;
        }
    }
    else
    {
        return -EINVAL;
    }

    return 0;
}

/**
 * Send signals to registered PID's.
 *
 * @return
 */
static void WakeupWaitQueue(GpioEvent_t Event)
{
    int i;

    /* some profile code */

    CurrentEvent = Event;

    // we dont wake up debug app hence "< MAX_GPIO_INT_CONSUMERS" is used in for loop
    for (i = 0; i < MAX_GPIO_INT_CONSUMERS; i++)
    {
        EventPresent[i] = 1;
    }
    wake_up_interruptible(&wait);
}

/**
 * 1ms Interrupt handler
 *
 * @return
 */
static irqreturn_t gpio_int_handler_1ms(int irq, void *irq_arg)
{
    uint64_t reg_read, my_core_num;
    unsigned long flags;
    GpioEvent_t event = KERN_NO_EVENT;

    /* code to clear the interrupt registers */


    /************ profiling start************/
    NowCycle = get_cpu_cycle();
    GpioIntProfilePtr->TotalOneMsInterrupts++;

    /* Check the max diff between consecutive interrupts */
    if (PrevCycle)
    {
        DiffOne = NowCycle - PrevCycle;
        if (DiffOne > GpioIntProfilePtr->OneMsMaxDiff)
            GpioIntProfilePtr->OneMsMaxDiff = DiffOne;
    }
    PrevCycle = NowCycle;

    TimeStamp.OneMsCount++; /* increment the counter */

    /* Store the timestamp */

    GpioIntProfilePtr->Gpio1msTimeStamp[GpioIntProfilePtr->IndexOne] = NowCycle;
    TimeStamp.OneMsTimeStampAtIsr = NowCycle;
    GpioIntProfilePtr->IndexOne++;
    if (GpioIntProfilePtr->IndexOne == GPIO_PROFILE_ARRAY_SIZE)
        GpioIntProfilePtr->IndexOne = 0;
    /************ profiling end************/

    /*
     * Whenever 10ms Interrupt happens we send only one event to the upper layers.
     * Hence it is necessary to sync between 1 & 10ms interrupts.
     * There is a chance that sometimes 1ms can happen at first and sometimes 10ms.
     *
     */
    /******** Sync mechanism ***********/
    spin_lock_irqsave(&GpioSyncLock, flags);    // Take the spinlock
    OneMsCounter++;
    OneMsTime = NowCycle;
    DiffOne = OneMsTime - TenMsTime;

    if (DiffOne < MAX_OFFSET_BETWEEN_1_AND_10MS)    //ten ms has happened first
    {
        if (OneMsCounter == 10)
        {
            event = KERN_BOTH_EVENT;
            SyncStarted = 1;
        }
        else
        {
            if (SyncStarted)
            {
                if (OneMsCounter < 10)
                {
                    GpioIntProfilePtr->TickSyncErrAt1msLess++;
                }
                else if (OneMsCounter > 10)
                {
                    GpioIntProfilePtr->TickSyncErrAt1msMore++;
                }
            }
        }
        OneMsCounter = 0;
    }
    else
    {
        if (OneMsCounter < 10)
        {
            if (SyncStarted)
            {
                event = KERN_ONE_MS_EVENT;
            }
        }
        else if (OneMsCounter > 10)
        {
            OneMsCounter = 0;
            if (SyncStarted)
            {
                GpioIntProfilePtr->TickSyncErrAt1msMore++;
            }
        }
    }
    TimeStamp.SFN = OneMsCounter;
    spin_unlock_irqrestore(&GpioSyncLock, flags);
    /******** Sync mechanism ***********/

    if(event != KERN_NO_EVENT)
        WakeupWaitQueue(event);

    OneMsIsrTime = get_cpu_cycle() - NowCycle;
    if (GpioIntProfilePtr->Max1msIsrTime < OneMsIsrTime)
        GpioIntProfilePtr->Max1msIsrTime = OneMsIsrTime;
    return IRQ_HANDLED;
}

/**
 * 10ms Interrupt handler
 *
 * @return
 */
static irqreturn_t gpio_int_handler_10ms(int irq, void *irq_arg)
{
    uint64_t reg_read, my_core_num;
    unsigned long flags;
    GpioEvent_t event = KERN_NO_EVENT;

    /* clear the interrupt */

    /************ profiling start************/
    GpioIntProfilePtr->TotalTenMsInterrupts++;
    Now = get_cpu_cycle();
    if (Prev)
    {
        DiffTen = Now - Prev;
        if (DiffTen > GpioIntProfilePtr->TenMsMaxDiff)
            GpioIntProfilePtr->TenMsMaxDiff = DiffTen;
    }
    Prev = Now;
    TimeStamp.OneMsCount++; /* increment the counter */
    TimeStamp.TenMsCount++;
    GpioIntProfilePtr->Gpio10msTimeStamp[GpioIntProfilePtr->IndexTen] = Now;
    TimeStamp.TenMsTimeStampAtIsr = Now;
    //do_gettimeofday(&TimeOfDayAtIsr.TimeAt10MsIsr);
    GpioIntProfilePtr->IndexTen++;
    if (GpioIntProfilePtr->IndexTen == GPIO_PROFILE_ARRAY_SIZE)
        GpioIntProfilePtr->IndexTen = 0;
    /************ profiling end************/

    /******** Sync mechanism ***********/
    spin_lock_irqsave(&GpioSyncLock, flags);
    TenMsTime = Now;
    DiffTen = TenMsTime - OneMsTime;

    if (DiffTen < MAX_OFFSET_BETWEEN_1_AND_10MS)    //one ms has happened first
    {
        if (OneMsCounter == 10)
        {
            TimeStamp.OneMsTimeStampAtIsr = Now;
            event = KERN_BOTH_EVENT;
            SyncStarted = 1;
        }
        OneMsCounter = 0;
    }
    else
    {
        if (SyncStarted)
        {
            if (OneMsCounter < 9)
            {
                GpioIntProfilePtr->TickSyncErrAt10msLess++;
                OneMsCounter = 0;
            }
            else if (OneMsCounter > 9)
            {
                GpioIntProfilePtr->TickSyncErrAt10msMore++;
                OneMsCounter = 0;
            }
        }
        else
        {
            if (OneMsCounter != 9)
                OneMsCounter = 0;
        }
    }
    TimeStamp.SFN = OneMsCounter;
    spin_unlock_irqrestore(&GpioSyncLock, flags);
    /******** Sync mechanism ***********/

    if(event != KERN_NO_EVENT)
        WakeupWaitQueue(event);

    TenMsIsrTime = get_cpu_cycle() - Now;
    if (GpioIntProfilePtr->Max10msIsrTime < TenMsIsrTime)
        GpioIntProfilePtr->Max10msIsrTime = TenMsIsrTime;

    return IRQ_HANDLED;
}

最佳答案

重置 EventPresent 在 wait_event_interruptible() 中等待事件 

EventPtr = &EventPresent[Instance];
...
status = wait_event_interruptible(wait, (*EventPtr == 1));
*EventPtr = 0;

看起来可疑。

如果WakeupWaitQueue()会并发执行，那么事件的设置

for (i = 0; i < MAX_GPIO_INT_CONSUMERS; i++)
    {
        EventPresent[i] = 1;
    }
wake_up_interruptible(&wait);

会丢失。

最好为引发的事件和已处理的事件设置两个独立的计数器:

uint64_t EventPresent[MAX_GPIO_INT_CONSUMERS]; // Number if raised events
uint64_t EventProcessed[MAX_GPIO_INT_CONSUMERS]; // Number of processed events

在这种情况下，条件可能是这些计数器的比较:

Gpio_compat_ioctl()
{
    ...
    EventPresentPtr = &EventPresent[Instance];
    EventProcessedPtr = &EventProcessed[Instance];
    ...
    status = wait_event_interruptible(wait, (*EventPresentPtr != *EventProcessedPtr));
    (*EventProcessedPtr)++;
    ...
}

WakeupWaitQueue()
{
    ...
    for (i = 0; i < MAX_GPIO_INT_CONSUMERS; i++)
    {
        EventPresent[i]++;
    }
    wake_up_interruptible(&wait);
}

关于c - 是不是内核卡住，我们在Stack Overflow上找到一个类似的问题： https://stackoverflow.com/questions/40809576/

上一篇：c - 使用 C 读取和写入整数到二进制文件？

下一篇：C:元素个数奇数的数组归并排序

相关文章：

c - 图的边

c - 在 C 中扫描整数后扫描字符数组

为内核模块创建 DKMS 包，依赖于内核头文件

linux - PCI/PCIe 设备如何在 Linux 内核中初始化/注册自己？

c - I2C 设备 linux 驱动程序

c - 从特定格式的文本文件中读取

c - 如何提高多线程读取文件的性能？

c++ - 退出全屏模式后清除屏幕？

linux-kernel - 获取通过引导加载程序 (grub) 传递的内核参数

linux - 中断处理程序中的 Printk 或 I/O 危险

©2024 IT工具网  联系我们