reactjs - 使用 Tensorflowjs 对上传的图像进行预测时，React 状态总是落后一步

Question

我正在开发一个 React 应用程序，使用 Tensorflow.js 根据图像对 Pokemon 进行分类。

• 我想要什么 - 上传 Pokemon 的图像，为同一 Pokemon 生成预测。

• 实际发生的情况 - 当我上传图像进行预测时，输出始终是前一张图像。所以，第一个预测总是垃圾(随机口袋妖怪)。当我上传第二个神奇宝贝时得到的预测始终是针对第一个神奇宝贝的。上传后的第三个神奇宝贝会预测第二个神奇宝贝，依此类推。

请参阅问题底部，了解其在何处有效以及在何处无效的详细版本。

以下是相关代码片段 -

1. 我首先检查模型是否存在于 indexeddb 中，如果是，我将其加载到状态 model 中。如果没有，我从服务器获取它并将其存储在状态中。这就是第一个 useEffect 在页面第一次呈现时所做的事情。

2. 我使用另一个 useEffect，只要 findState.uploadedImage 发生变化，它就会运行。 Redux-toolkit 中存在此状态。

这是问题的简短演示 => https://youtu.be/MX70zbupNWQ

这是应用程序 URL => https://poke-zoo.herokuapp.com/

这是 Github 存储库 => https://github.com/theairbend3r/poke-zoo/tree/master/frontend/src/features/find

这是文件SearchOutput.js。这将获取模型并进行预测。

const SearchOutput = () => {
  const findState = useSelector(selectorFind)
  const dispatch = useDispatch()
  const imageRef = useRef(null)

  const [model, setModel] = useState(null)
  const [predictions, setPredictions] = useState([])

  const MODEL_HTTP_URL = "api/pokeml/classify"
  const MODEL_INDEXEDDB_URL = "indexeddb://poke-model"

  useEffect(() => {
    async function fetchModel() {
      try {
        const localClassifierModel = await tf.loadLayersModel(
          MODEL_INDEXEDDB_URL
        )

        setModel(localClassifierModel)
        console.log("Model loaded from IndexedDB")
      } catch (e) {
        const classifierModel = await tf.loadLayersModel(MODEL_HTTP_URL)
        setModel(classifierModel)

        await classifierModel.save(MODEL_INDEXEDDB_URL)

        console.error(e)
      }
    }
    fetchModel()
  }, [])

  const getTopKPred = (pred, k) => {
    const predIdx = []
    const predNames = []

    const topkPred = [...pred].sort((a, b) => b - a).slice(0, k)

    topkPred.map(i => predIdx.push(pred.indexOf(i)))
    predIdx.map(i => predNames.push(idx2class[i]))

    return predNames
  }

  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef.current)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127.5)
            .div(127.5)
            .expandDims()

          const y_pred = await model.predict(imgTensor).dataSync()
          const topkPredNames = getTopKPred(y_pred, 5)

          console.log(topkPredNames)
          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.")
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage])

  return (
     <div>
        {findState.uploadedImage && (
          <img
            ref={imageRef}
            tw="border border-purple-700 p-1 rounded shadow-lg"
            src={findState.uploadedImage}
            width={600}
            height={600}
          />
        )}


        <div>
           {findState.matchesFound.length === 6 &&
             findState.matchesFound.map(poke => (
               <PokemonCardML
                 key={`key-${poke.id}`}
                 pokemonId={poke.id}
                 pokemonName={poke.name}
                 pokemonType={poke.type}
                 pokemonHeight={poke.height}
                 pokemonWeight={poke.weight}
                 pokemonBaseExperience={poke.baseExperience}
                 pokemonSprite={poke.sprites}
               />
             ))}
       </div>
     </div>
)
}

这是将输入图像存储到 redux 状态的文件 findSlice.js。

import { createSlice } from "@reduxjs/toolkit"
import axios from "axios"

const initialState = {
  uploadedImage: "",
  model: null,
  matchesFound: [],
}

export const findSlice = createSlice({
  name: "find",
  initialState: initialState,
  reducers: {
    storeInputImage: (state, action) => {
      state.uploadedImage = action.payload.uploadedImage
    },
    setModel: (state, action) => {
      state.model = action.payload.model
    },
  },
})

export const selectorFind = state => state.find
export const { storeInputImage, setModel } = findSlice.actions
export default findSlice.reducer

---

有关问题的详细信息。

### Desktop

#### Table

|         |   Ubuntu    |     Windows      |      MacOS       |
| ------- | :---------: | :--------------: | :--------------: |
| Firefox | not working |   not working    |   not working    |
| Chrome  | not working | somewhat working | somewhat working |
| Safari  |     NA      |        NA        | somewhat working |

#### Comments

|         |                                    Ubuntu                                    |                                   Windows                                    |                                    MacOS                                     |
| :-----: | :--------------------------------------------------------------------------: | :--------------------------------------------------------------------------: | :--------------------------------------------------------------------------: |
| Firefox | Predictions are always one step behin for both Captured and Uploaded images. | Predictions are always one step behin for both Captured and Uploaded images. | Predictions are always one step behin for both Captured and Uploaded images. |
| Chrome  | Works only on Captured Images. Uploaded images give same predictions always. | Works only on Captured Images. Uploaded images give same predictions always. | Works only on Captured Images. Uploaded images give same predictions always. |
| Safari  |                                      NA                                      |                                      NA                                      | Works only on Captured Images. Uploaded images give same predictions always. |

### Mobile

#### Table

|         |     Android      |       iOS        |
| ------- | :--------------: | :--------------: |
| Firefox | somewhat working |   not working    |
| Chrome  | somewhat working |   not working    |
| Safari  |        NA        | somewhat working |

#### Comments

|         | Android                                                                     | iOS                                                                          |
| :-----: | --------------------------------------------------------------------------- | ---------------------------------------------------------------------------- |
| Firefox | No predictions load for a captured image. Works on uploaded images only.    | Camera does not load. Uploaded images give same predictions always.          |
| Chrome  | Works on capture images only. Uploaded images give same predictions always. | Camera does not load. Uploaded images give same predictions always.          |
| Safari  | NA                                                                          | Works only on Captured Images. Uploaded images give same predictions always. |

---

编辑:基于以下建议。但这并没有解决问题。放在这里供引用。

const SearchOutput = () => {
  const findState = useSelector(selectorFind)
  const [imageRef, setImageRef] = useState(null)

  const onChangeRef = useCallback(node => {
    setImageRef(node)
  }, [])

  const [model, setModel] = useState(null)
  const [predictions, setPredictions] = useState([])

  const MODEL_HTTP_URL = "api/pokeml/classify"
  const MODEL_INDEXEDDB_URL = "indexeddb://poke-model"


  useEffect(() => {
    async function fetchModel() {
      try {
        const localClassifierModel = await tf.loadLayersModel(
          MODEL_INDEXEDDB_URL
        )

        setModel(localClassifierModel)
        console.log("Model loaded from IndexedDB")
      } catch (e) {
        try {
          const classifierModel = await tf.loadLayersModel(MODEL_HTTP_URL)
          setModel(classifierModel)

          await classifierModel.save(MODEL_INDEXEDDB_URL)
          console.log("Model saved to IndexedDB")
        } catch (e) {
          console.log("Unable to load model at all: ", e)
        }
      }
    }
    fetchModel()
  }, [])

  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        console.log(
          "Uploaded Image from inside the useEffect",
          findState.uploadedImage
        )
        console.log("ImageRef from inside the useEffect", imageRef.current)
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef.current)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127)
            .div(127)
            .expandDims()

          const y_pred = await model.predict(imgTensor).data()
          console.log(y_pred)
          console.log(pokemonState)

          const topkPredNames = getTopKPredPokeObj(y_pred, 6, pokemonState)

          dispatch(storePredictions({ predictions: topkPredNames }))

          console.log(topkPredNames)

          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.", e)
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage])

  return (
    <div>
          {findState.uploadedImage && (
            <img
              ref={onChangeRef}
              src={findState.uploadedImage}
              width="600"
              height="600"
            />
          )}
    </div>
  )
}

export default SearchOutput

Answer 1

我之前读过这个问题，但我自己没有遇到过这个问题，但这应该对您有帮助。

The useRef hook can be a trap for your custom hook, if you combine it with a useEffect that skips rendering. Your first instinct will be to add ref.current to the second argument of useEffect, so it will update once the ref changes. But the ref isn’t updated till after your component has rendered — meaning, any useEffect that skips rendering, won’t see any changes to the ref before the next render pass.

正如您在遇到的行为中所看到的，imageRef.current在你的useEffect里面不返回记录先前值的更新对象。

解决此问题的合理方法是执行以下操作:

  useEffect(() => {
    async function makePredictions() {
      //...
    }
    makePredictions()
  }, [findState.uploadedImage, imageRef.current])

但问题是 imageRef.current 发生了变化不会触发 React 中的渲染。因此，根据有关如何测量 DOM 节点的 React 文档，您应该使用 useCallback而不是useRef .

这样的事情应该有效:

  const [imageRef, setImageRef] = useState(null);
  const onChangeRef = useCallback(node => {
    // ref value changed to node
    setImageRef(node); // e.g. change ref state to trigger re-render
    if (node === null) { 
      // node is null, if DOM node of ref had been unmounted before
    } else {
      // ref value exists
    }
  }, []);
  
  useEffect(() => {
    async function makePredictions() {
      if (imageRef && model) {
        try {
          const imgTensor = tf.browser
            .fromPixels(imageRef)
            .resizeNearestNeighbor([160, 160])
            .toFloat()
            .sub(127.5)
            .div(127.5)
            .expandDims()

          const y_pred = await model.predict(imgTensor).dataSync()
          const topkPredNames = getTopKPred(y_pred, 5)

          console.log(topkPredNames)
          return topkPredNames
        } catch (e) {
          console.log("Unable to run predictions.")
        }
      }
    }
    makePredictions()
  }, [findState.uploadedImage, imageRef])

而不是 <img ref={imageRef} /> ，你应该使用<img ref={onChangeRef} /> .

引用文献:

Ref objects inside useEffect Hooks
How can I measure a DOM node?