cannot open the pkl file

[yaoli]

File "sample.py", line 1, in
import cPickle as pkl
File "sample.py", line 65, in main
tasks = pkl.load(fin)
UnpicklingError: invalid load key, 'v'.

[ffmpbgrnn]

Hi Li, can you please check the md5 of pickle files in the datasets directory which should be like:

08905c5f08f43b86abc65644a9955b2f  MED_easy.pkl
cb16a7768b82310388a8744e07ab0943  MED_hard.pkl
dd3523207f7d17d4e5a10f9c392903f9  MPII_easy.pkl
be712f8b8eaa278f9ac13fbea904fbdf  MPII_hard.pkl
fb68111257defb82fc761471a9c79a18  TACoS_easy.pkl
87fe81234f8f57679a399df1aa89a2d1  TACoS_hard.pkl
ec486bb522a31f0d3d091303184ef7dc  meta_tasks.pkl

you can just check by typing md5sum *.

I just downloaded the dataset again and the sample.py runs smoothly.

Did you install git-lfs and initialize it? If not, you should go to https://github.com/github/git-lfs and install it and then initialize with git lfs install. Now you can do normal git clone

[yaoli]

I didn't install git-lfs, not sure what it's actually doing. Why does loading pkl fail without it?
I'm using a Linux machine.

[ffmpbgrnn]

git-lfs is used for large files. The pkl files are not stored in the original repository, but are tracked by git lfs.
So you need git-lfs to download the pkl files. If you don't want to use git-lfs, you can download them manually, for example, go to https://github.com/ffmpbgrnn/VideoQA/blob/master/datasets/MED_easy.pkl and click raw.

[ffmpbgrnn]

If you cat the output of pkl files you clone, you can see something like

version https://git-lfs.github.com/spec/v1
oid sha256:acd317d3d403a2d2cee46f0b3e3bdde142266c2ac14e613b12ceb6f75d102666
size 25928870

As can be seen, it is only a pointer to the original file.

[yaoli]

Oh, I see. That's why. Thanks a lot. I'll try it later.

[yaoli]

It works. Thanks a lot.
From what I can see in those pkl files, videos on TACoS are further chopped into small short clips. To be consistent, it would be super handy for us to use your script to chop them and assign the same video IDs used in those pkl files. I hope we are not asking for too much.

[ffmpbgrnn]

Hi Li, if you downloaded TACoS annotations, there is a index.tsv file which defines the correspondence between clip name and the actual frames in the video. For example,

s37-d46_19_7›   The man takes the slices of pineapple and places them on top of each other to nicely dice them into squares.›   s37-d46›6645›   11145›  pineapple›  11196

it shows the clip s37-d46_19_7 corresponds to the video s37-d46.avi between frame 6645 and 11145.

I simply use ffmpeg to extract all frames rather than chop the whole video into small clips.

[yaoli]

Thanks very much indeed.

[ffmpbgrnn]

No worries.

[yaoli]

{'questions': [{'T': 'person put pan on _ oven .', 'W3': 'leek', 'W2': 'start', 'W1': 'unwrap', 'W0': 'heat'}, {'T': 'person put _ on heat oven .', 'W3': 'proceed', 'W2': 'pull out', 'W1': 'bowl', 'W0': 'pan'}, {'T': 'person _ pan on heat oven .', 'W3': 'attempt', 'W2': 'open up', 'W1': 'enlarge', 'W0': 'put'}, {'T': 'person _ heat oven .', 'W3': 'cut off top of pepper', 'W2': 'get cut board from drawer', 'W1': 'cut slice in half', 'W0': 'put pan on'}, {'T': 'person put pan on heat _ .', 'W3': 'tap', 'W2': 'room', 'W1': 'rag', 'W0': 'oven'}, {'T': 'person put _ on heat oven .', 'W3': 'stone', 'W2': 'faucet', 'W1': 'frame', 'W0': 'pan'}], 'desc': u'Person puts pan on heated oven.'}

This example is a bit strange as the blanks are phrases, instead of single words. Are this kind of QA pairs used at all in training?

Here are more of answer candidates from the entire dataset, the number indicate number of appearance:

['throw end into wastebin', 1], ['procedure', 11], ['possible', 11], ['remove fry pan from drawer', 1], ['cut cucumber in half', 1], ['add water to bowl', 1], ['take onion from pantry', 1], ['put broccoli in bowl', 3], ['juices first half of lime', 1], ['pour into pan', 3], ['discarding', 4], ['cube', 2], ['widest', 3], ['pine', 11], ['put pineapple on plate', 1], ['get cut board out of drawer', 1], ['take out second bowl', 1], ['put in tupperware container', 2], ['get out bowl', 1], ['cut off top of pepper', 2], ['cut slice in half', 1], ['get out leek', 3], ['throw leave into trash', 2], ['corn', 3], ['starch', 3], ['3/4', 12], ['smile', 3], ['lovely', 3], ['monitor', 4], ['commence', 10], ['assort', 13], ['put on table', 1], ['till', 34], ['successfully', 5], ['re-stirs', 19], ['pours', 11], ['add spice to egg', 1], ['cut in four piece', 2], ['30', 6], ['sort', 10], ['place in pot', 1], ['select knife from drawer', 2], ['render', 7], ['pee', 8], ['washer', 9], ['15', 14], ['put fig on plate', 3], ['get knife cut board', 1], ['get out spoon stir ginger', 1], ['facet', 21], ['wash knife cut board', 3], ['take package of sugar from pantry', 2], ['unbundles', 7], ['set on countertop', 3], ['procure egg from fridge', 2], ['satisfactory', 4], ['move piece to plate', 1], ['consumption', 6], ['relocate', 4], ['put orange on plate', 1], ['cut broad bean with', 1], ['pour juice into glass', 1], ['swipe', 23], ['uncooked', 11], ['put piece in bowl', 1], ['cubbord', 6], ['take out spoon from drawer', 4], ['cut fig in half', 1], ['cut off stem', 2], ['skin second potato', 1], ['chop ginger on cut board', 2], ['return cut board to drawer', 1], ['remove stem seed', 1], ['put cut board away', 1], ['!!!', 9], ['sample', 3], ['fire', 3], ['take out pot fill with water', 2], ['cut off tip of bean throw away', 1], ['naked', 14], ['For', 11], ['place cut board on counter', 1], ['break orange into quarter', 1], ['return to cabinet', 1], ['throughly', 8], ['refrigerate', 2], ['cut pomegranate in half', 2], ['soften', 5], ['wash cut board knife in sink', 1], ['repackages', 6], ['unsliced', 6], ['offscreen', 9], ['herbes', 3], ['reason', 11], ['decides', 11], ['take out lime', 1], ['take out round cut board', 3], ['poor', 7], ['vigorously', 13], ['stirring', 3], ['Saute', 3], ['remove cut board from drawer', 1], ['nine', 3], ['using', 7], ['paring', 7], ['shallow', 13], ['take out cut board knife place on counter', 2], ['throw away waste', 2], ['return package of sugar to pantry', 3], ['disperse', 18], ['electric', 16], ['ignite', 16], ['reminisce', 14], ['chop up chili', 2], ['spiral', 5], ['simultaneously', 5], ['energetically', 4], ['ease', 5], ['9', 3], ['triangles', 8], ['grease', 6], ['chop ginger put on plate', 1], ['swish', 6], ['grain', 9], ['open up drawer', 1], ['get large knife from drawer', 2], ['generous', 4], ['put piece of leek on plate', 1], ['place knife on cut board', 1], ['follow', 6], ['partway', 11], ['wipe down counter with rag', 1], ['dumps', 15], ['gracious', 15], ['slice across cut', 1], ['deftly', 8], ['add broccoli to pot', 1], ['slice onion in half', 1], ['lenghwise', 4], ['mark', 8], ['get peeler out of drawer', 2], ['extract juice', 1], ['kiwifruit', 11], ['opt', 8], ['inside/outside', 11], ['effieciently', 10], ['wither', 2], ['wisk', 5], ['sized', 3], ['is', 6], ['extracted', 6], ['Before', 5], ['remove avocado from refrigerator', 1], ['peel half of kiwi', 1], ['difficult', 14], ['wait for cauliflower to cook', 1], ['pressure', 5], ['deeper', 8], ['freshness', 10], ['slice fig on cut board', 1], ['get cut board from cupboard', 1], ['chip', 5], ['expose', 20], ['knife to peel orange over sink', 1], ['take out figs', 2], ['melted', 5], ['get out wooden spatula', 1], ['unsavory', 4], ['uncover', 7], ['throw away bottom of cauliflower', 1], ['rinse dry plum', 2], ['care', 6], ['put away spoon', 1], ['semicircle', 9], ['semicircles', 4], ['capture', 7], ['wash broad bean in sink', 1], ['broccolli', 3], ['break orange in half', 1], ['remove from refrigerator', 1], ['snall', 3], ['meticulously', 7], ['bisect', 10], ['add olive oil to pan', 1], ['rinse under faucet', 1], ['weaken', 8], ['slotted', 5], ['metallic', 13], ['escape', 20], ['walk to cabinet', 1], ['home', 4], ['cupboards', 5], ['straightens', 2], ['lids', 7], ['orange-half', 15], ['back-and-forth', 12], ['wash knife in sink', 2], ['mid-way', 9], ['navel', 9], ['throw away seed', 1], ['oddly', 10], ['3rd', 7], ['painstakingly', 4], ['unedible', 4], ['add to pot', 1], ['board/plate', 4], ['take out metal bowl from drawer', 1], ['magic', 7], ['massage', 10], ['Grates', 4], ['inspection', 3], ['get out wooden spatula stir oil', 1], ['add carrots to pan', 1], ['put rest away', 1], ['rinse off in sink', 1], ['throw away outside of pomegranate', 1], ['smell', 2], ['peel first kiwi', 1], ['lip', 11], ['wash put away', 1], ['think', 11], ['shakers', 15], ['get out fig', 1], ['many', 3], ['add salt pepper to leeks', 1], ['int', 8], ['exit', 5], ['peel/scoop', 10], ['sheaf', 7], ['Using', 8], ['take bottle of oil from pantry', 1], ['return half to fridge', 1], ['dinner', 4], ['fine-ness', 3], ['disposes', 3], ['twirl', 10], ['periodically', 9], ['During', 9], ['rubbish', 8], ['therefore', 3], ['softer', 3], ['lenghtwise', 2], ['smack', 6], ['dirt', 7], ['kictchen', 2], ['set on cut board', 1], ['contemplate', 11], ['directions', 6], ['shoot', 5], ['resume', 12], ['seal/wrapper', 4], ['decent', 8], ['actually', 6], ['add to water', 1], ['bone', 6], ['similar-sized', 4], ['cut off top bottom of pineapple', 1], ['dozen', 5], ['whip', 3], ['scoop out potatoes', 1], ['take small glass from cupboard', 1], ['get out small plate', 1], ['chop up cucumber', 1], ['split in half', 1], ['cut end off of pomegranate', 1], ['ninty', 9], ['rewrapped', 3], ['hald', 6], ['get out broccoli', 1], ['stemless', 6], ['chop bean put in bowl', 1], ['get out plum', 1], ['unusable/spoiled', 4], ['determine', 3], ['appropriately', 3], ['butter-flour', 4], ['spice/flavoring', 6], ['creamsauce', 3], ['damaged/spoiled', 15], ['central', 7], ['pod', 3], ['cylindrical', 4], ['seal', 4], ['put slice on plate', 1], ['put pepper on plate', 1], ['continuously', 10], ['overcook', 10], ['catsup', 4], ['throw out shell', 1]]

[ffmpbgrnn]

The answers can be phrases, like put on, take off, cut off etc, however, these phrases have much lower frequencies than single word.