Tech review of AI Plank Tutor LP#3470
Conversation
| Run the app with real camera input. Check whether the model gives short, useful corrections, whether latency is acceptable, and whether repeated prompts produce varied but consistent feedback. | ||
|
|
||
| Fine-tuning is not included as a formal step in this Learning Path because it adds dataset design, training infrastructure, model conversion, quantization, and evaluation. However it will make a significant quality difference to the app, as you get much better responses. Treat it as a follow-up project once the app pipeline is working. | ||
| Fine-tuning is not included as a formal step in this Learning Path because it adds dataset design, training infrastructure, model conversion, quantization, and evaluation. However, it can make a significant quality difference to the app by making responses more consistent. Treat it as a follow-up project once the app pipeline is working. |
There was a problem hiding this comment.
In this app it's less about the added consistency, and more about sounding like a yoga teacher, better form of advice (eg "Extend your arms" vs "Straighten right elbow") and accuracy. Unsure of "in character" or "true to the persona" or if you have a better term.
Fine-tuning is not included as a formal step in this Learning Path because it adds dataset design, training infrastructure, model conversion, quantization, and evaluation. However, it can make a significant quality difference to the app by making responses more accurate and true to the persona. Treat it as a follow-up project once the app pipeline is working.
BmanClark
left a comment
BmanClark
left a comment
There was a problem hiding this comment.
Just a few comments. Working with Arnaud on one last license thing, then demo code should get merged into main.
| ## Clone the starter project | ||
|
|
||
| Clone the Learning Path code examples repository: | ||
| Clone the `PlankTutor` branch of the Learning Path code examples repository. This branch contains the starter Android project used by this Learning Path: |
There was a problem hiding this comment.
Hopefully once accepted it will be merged in, so won't need branch.
|
|
||
| MediaPipe landmarks are normalized coordinates. The `x` and `y` values are relative to the input image, and `z` gives relative depth. | ||
|
|
||
| Because this demo compares normalized landmarks from a fixed instructor reference, the score is view-dependent. It works best when the learner is side-on to the camera, fully visible, and at a similar orientation to the reference image. |
There was a problem hiding this comment.
I'm unsure about the first sentence. In theory the normalized landmarks are trying to make it more view-independent, give a 3D representation. In reality they only work so well and the second sentence is true though.
Maybe:
"Although this demo compares normalized landmarks from a fixed instructor reference, the score is still fairly view-dependent." ?
|
|
||
| This keeps the prompt small and gives the model only the facts it needs to produce one coaching cue. | ||
|
|
||
| This is the main design pattern in the app: the vision model turns camera frames into structured pose data, deterministic Kotlin code turns that data into a small set of facts, and the LLM only handles the language generation step. |
There was a problem hiding this comment.
The LLM is doing a little bit more than that, it's turning that small set of facts into appropriate advice.
"and the LLM takes these facts and returns appropriate advice." ?
|
|
||
| {{% notice Note %}} | ||
| The sign of the difference is important. A positive value means the learner needs to straighten that joint compared with the reference. A negative value means the learner needs to bend it more. | ||
| The sign of the difference is important. `difference = reference - learner`, so a positive value means the learner's angle is smaller than the reference and usually needs to straighten. A negative value means the learner's angle is larger than the reference and usually needs to bend more. |
There was a problem hiding this comment.
I don't think we need the "usually"s.
| ``` | ||
|
|
||
| `AiChat.getInferenceEngine()` creates the inference engine used by the ViewModel. `LlmModelStore` is the helper that finds or imports the GGUF model file. Now we need to make the `LlmViewModel` accept those parameters. | ||
| `AiChat.getInferenceEngine()` creates a native-backed inference engine used by the ViewModel. Reuse this engine for the app session instead of creating one for every prompt. `LlmModelStore` is the helper that finds or imports the GGUF model file. Now make the `LlmViewModel` accept those parameters. |
There was a problem hiding this comment.
"Reuse this engine for the app session instead of creating one for every prompt."
Seems a weird thing to have to mention? Can keep, but feels unnecessary.
| Open Logcat and filter for `MainActivity`. As you move in front of the camera, you should see short prompts that describe the largest differences from the reference plank pose. | ||
| Open Logcat and filter for `MainActivity`. As you move in front of the camera, expect short prompts that describe the largest differences from the reference plank pose. | ||
|
|
||
| This Logcat check is the validation step for the prompt stage. Expect compact text prompts, not raw landmarks or image data. |
There was a problem hiding this comment.
We've already said in the previous sentence to expect short prompts, second sentence unnecessary?
2 participants
Before submitting a pull request for a new Learning Path, please review Create a Learning Path
Please do not include any confidential information in your contribution. This includes confidential microarchitecture details and unannounced product information.
By submitting this pull request, I confirm that you can use, modify, copy, and redistribute this contribution, under the terms of the Creative Commons Attribution 4.0 International License.